Hi again,
One may want to have the manager-process receive and loop through @sample and @good. That will incur an additional CPU core for the manager-process itself.
use strict;
use warnings;
use MCE;
open my $sample_fh, ">", "sample.txt" or die "open error: $!";
open my $good_fh, ">", "good.txt" or die "open error: $!";
# worker function
sub task {
my ( $mce, $slurp_ref, $chunk_id ) = @_;
my ( @sample, @good );
# open file handle to scalar ref
open my $input_fh, "<", $slurp_ref;
# append to scalars inside the loop
while (<$input_fh>) {
if (/^sample\s+(\S+)/) {
push @sample, $1;
}
elsif (/^good\s+(\S+)/) {
push @good, $1;
}
}
close $input_fh;
# send arrays to the manager-process
MCE->gather(\@sample, \@good);
}
# manager function
sub gather {
my ( $sample, $good ) = @_;
# process sample
for ( @{ $sample } ) {
;
}
# process good
for ( @{ $good } ) {
;
}
}
# spawn workers early, optionally
my $mce = MCE->new(
chunk_size => '1m', # 1 megabyte
max_workers => 4,
use_slurpio => 1,
user_func => \&task,
gather => \&gather,
)->spawn;
# process input file(s)
$mce->process({ input_data => "test.txt" });
# shutdown workers
$mce->shutdown;
# close output handles
close $sample_fh;
close $good_fh;
The extra time comes from workers appending to local arrays. Likewise, the manager-process receiving and looping through the arrays. There are 4 workers and the manager process running simultaneously on a machine with 4 real cores.
$ time perl test_demo.pl
real 0m9.932s
user 0m43.956s
sys 0m0.452s
Update:
Interestingly, Perl v5.20 and higher take 2x longer to run. I'm not sure why. Yikes, possibly from regular expression? This is on my TODO list to check why. The above was captured from Perl v5.18.2 on the same machine.
$ time /opt/perl-5.20.3/bin/perl test_demo.pl
real 0m20.858s
user 1m20.164s
sys 0m8.488s
Regards, Mario