Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

comment on

( #3333=superdoc: print w/replies, xml ) Need Help??
Felow monks,

Recently I was working a lot with threads, producing a group of services that should work 24h per day, 7 days per week. This services are multithread since they access the internet doing some tasks, so they need to split the tasks in many threads to do the job faster and use better the hardware and link of our servers. But we have the problem that each thread in Perl uses more and more memory, so I want to share my experience on trying to fix that.

To do this tasks over the internet we use WWW::Mechanize that make our life much more easy, specially when we need to login in some site.

Today Perl 5.8.6 is a real solution for multithread services, but we have a big problem, memory usage. Our servers don't run only one service, but many services, and each thread of WWW::Mechanize uses a lot of memory! So, we can't start much more than 5 threads per process, since with 5 threads the service uses 60Mb+-!

Looking for a solution I have created Thread::Isolate, where the main idea is to create a thread than can be called externally, so, I can load modules from outside and call them. With that I can organize my programs in some threads, each of them with different resources and only duplicate threads/resources that will need to run in parallel.

Here's a simple sample of Thread::Isolate:

use Thread::Isolate ; my $thi = Thread::Isolate->new() ; $thi->use('Data::Dumper') ; print $thi->call('Data::Dumper::Dumper' , [123 , 456]) ;
Also I can do detached (unsynchronous) calls:
use Thread::Isolate ; $|=1 ; my $thi = Thread::Isolate->new() ; $thi->use('LWP::Simple'); my $job = $thi->call_detached('LWP::Simple::get' , 'http://www.perlm +onks.com/') ; while( $job->is_running ) { print "." ; sleep(1); } $html = $job->returned ; die( $thi->err ) if $thi->err ; print $html ;
Looking the code above we can see that we are not only isolating LWP::Simple, but we also can run a code until get() returns something.

Thread::Isolate was inspirated on Thread::Tie::Thread by Elizabeth Mattijsen (liz), that did something like that to make Thread::Tie. Than I saw that I could use that to organize my multithread programs, so I created a new module with more resources and an intuitive interface, enjoy! ;-P

Like any new module it still needs tests, and a module that works with multithread need more tests, so, I'm asking for you monks to test it and report your experience with it here. Thanks in advance.

CPAN should put it soon at:
http://www.linorg.usp.br/CPAN/authors/id/G/GM/GMPASSOS/

Graciliano M. P.
"Creativity is the expression of liberty".


In reply to Isolating dynamically loaded modules with Thread::Isolate. by gmpassos

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others wandering the Monastery: (7)
    As of 2020-07-14 14:36 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      No recent polls found

      Notices?