Hi again perlygapes,
The MCE::Loop code just abstracts away all your Parallel::ForkManager logic and improves it, just as Parallel::ForkManager abstracts away and improves some of the tedious manual work of using fork() directly. See how the logic is encapsulated in a sub just like in your code, only with less concurrency boilerplate.
"using a separate DB connection instead for each child feels intuitively right"
I agree, the code I shared keeps a connection open for each child, which itself stays alive and handles multiple jobs from the job list as managed by MCE.
Here's a simpler example I've shared recently showing how to parallelize existing code for making a series of HTTP requests. How would you do the same using P::FM?
single process
use strict; use warnings; use 5.010;
use Data::Dumper; use HTTP::Tiny;
use Time::HiRes 'gettimeofday', 'tv_interval';
my $ua = HTTP::Tiny->new( timeout => 10 );
my @urls = qw< gap.com amazon.com ebay.com lego.com wunderground.com
imdb.com underarmour.com disney.com espn.com dailymail.com >;
my %report;
foreach( @urls) {
my $start = [gettimeofday];
$ua->get('https://' . $_);
$report{$_} = tv_interval($start, [gettimeofday]) );
});
say Dumper \%report;
six processes
(workers stay alive, looping through the list, writing to a shared hash)
(one added line, two slightly changed lines)
use strict; use warnings; use 5.010;
use Data::Dumper; use HTTP::Tiny;
use Time::HiRes 'gettimeofday', 'tv_interval';
use MCE; use MCE::Shared;
my $ua = HTTP::Tiny->new( timeout => 10 );
my @urls = qw< gap.com amazon.com ebay.com lego.com wunderground.com
imdb.com underarmour.com disney.com espn.com dailymail.com >;
my $report = MCE::Shared->hash;
MCE->new( max_workers => 6 )->foreach( \@urls, sub {
my $start = [gettimeofday];
$ua->get('https://' . $_);
$report->set( $_, tv_interval($start, [gettimeofday]) );
});
say Dumper $report->export;
Update: fixed error in first demo code, ++choroba
Hope this helps!
The way forward always starts with a minimal test.
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.