comment on

Update: Added results for Parallel::ForkManager 2.02. The results mentioned in the POD documentation ran slower due to an unaware background job at the time. I reran again on all 5 platforms.

Dear fellow Monks,

I am pleased to annouce MCE 1.842 and MCE::Shared 1.842 (both stable). MCE now includes MCE::Channel and MCE::Child recently.

This weekend, added Parallel::ForkManager-like demonstration to the POD section in MCE::Child and MCE::Hobo. The results were captured on a Macbook Pro (late 2013 model - 2.6 GHz ~ 3.6 GHz with Turbo Boost).

To run, one may direct standard output to :nul or /dev/null depending on the platform. Or better yet, I commented out the print line in the on_finish handler. Unfortunately, Parallel::ForkManager 2.02 suffers from memory leaks on the Windows platform ($^O eq 'MSWin32') and the reason why slower than Cygwin.

Time to spin 2,000 workers and obtain results (in seconds).

MCE::Hobo uses MCE::Shared to retrieve data during reaping.
MCE::Child uses MCE::Channel, no shared-manager.

         Version  Cygwin   Windows  Linux   macOS  FreeBSD

MCE::Child 1.842  19.099s  17.091s  0.965s  1.534s  1.229s
 MCE::Hobo 1.842  20.514s  19.594s  1.246s  1.629s  1.613s
     P::FM 1.20   19.703s  19.235s  0.875s  1.445s  1.346s

MCE::Child 1.842  20.426s  18.417s  1.116s  1.632s  1.338s  Moo loaded
 MCE::Hobo 1.842  21.809s  20.810s  1.407s  1.759s  1.722s  Moo loaded
     P::FM 2.02   21.668s  25.927s  1.882s  2.612s  2.483s  Moo used
[download]

Set posix_exit to avoid all END and destructor processing.

MCE::Child->init( posix_exit => 1, ... );
 MCE::Hobo->init( posix_exit => 1, ... );

         Version  Cygwin   Windows  Linux   macOS  FreeBSD

MCE::Child 1.842  19.815s  ignored  0.824s  1.284s  1.245s  Moo loaded
 MCE::Hobo 1.842  21.029s  ignored  0.953s  1.335s  1.439s  Moo loaded
[download]

Parallel::ForkManager

use strict;
use warnings;

use Parallel::ForkManager;
use Time::HiRes 'time';

my $start = time;

my $pm = Parallel::ForkManager->new(10);
$pm->set_waitpid_blocking_sleep(0);

$pm->run_on_finish( sub {
    my ($pid, $exit_code, $ident, $exit_signal, $core_dumped, $resp) =
+ @_;
  # print "child $pid completed: $ident => ", $resp->[0], "\n";
});

DATA_LOOP:
foreach my $data ( 1..2000 ) {
    # forks and returns the pid for the child
    my $pid = $pm->start($data) and next DATA_LOOP;
    my $ret = [ $data * 2 ];

    $pm->finish(0, $ret);
}

$pm->wait_all_children;

printf STDERR "duration: %0.03f seconds\n", time - $start;
[download]

MCE::Child

use strict;
use warnings;
 
use MCE::Child 1.842;
use Time::HiRes 'time';
 
my $start = time;
 
MCE::Child->init(
    max_workers => 10,
    on_finish   => sub {
        my ($pid, $exit_code, $ident, $exit_signal, $error, $resp) = @
+_;
      # print "child $pid completed: $ident => ", $resp->[0], "\n";
    }
);
 
foreach my $data ( 1..2000 ) {
    MCE::Child->create( $data, sub {
        [ $data * 2 ];
    });
}
 
MCE::Child->wait_all;
 
printf STDERR "duration: %0.03f seconds\n", time - $start;
[download]

MCE::Hobo

use strict;
use warnings;
 
use MCE::Hobo 1.842;
use Time::HiRes 'time';
 
my $start = time;
 
MCE::Hobo->init(
    max_workers => 10,
    on_finish   => sub {
        my ($pid, $exit_code, $ident, $exit_signal, $error, $resp) = @
+_;
      # print "child $pid completed: $ident => ", $resp->[0], "\n";
    }
);
 
foreach my $data ( 1..2000 ) {
    MCE::Hobo->create( $data, sub {
        [ $data * 2 ];
    });
}
 
MCE::Hobo->wait_all;
 
printf STDERR "duration: %0.03f seconds\n", time - $start;
[download]

These days, there are other ways and instead have workers persist (i.e. pull items from a shared queue or channel) or perhaps via MCE's input and chunking capabilities.

Regards, Mario

In reply to New stable MCE 1.842 and MCE::Shared 1.842 releases by marioroy

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


P is for Practical
	PerlMonks