Re^2: Camel vs. Gopher

Go is cool. So is Perl. :) Below, demonstrations based on Your_Mother's example.

Serial Code

use strict;
use warnings;

use Time::HiRes 'time';

my $start = time;
my @ten = ( 0 .. 9 );
my %count;

$count{ $ten[ rand @ten ] }++ for 1..10_000_000;

print "$_ -> $count{$_}\n" for @ten;
printf "duration: %0.3f seconds\n", time - $start;
[download]

Parallel Implementation

Disclaimer: I typically use the OO interface for shared variables. The reason is exclusive locking handled automatically.

use strict;
use warnings;

use MCE;
use MCE::Shared;
use Time::HiRes 'time';

my $start = time;
my @ten   = ( 0 .. 9 );
my $count = MCE::Shared->hash();

MCE->new(
   max_workers => 4,
   sequence    => [ 1, 10_000_000 ],
   bounds_only => 1,
   chunk_size  => 50_000,

   user_func   => sub {
      my ( $mce, $seq, $chunk_id ) = @_;
      my %lcount;

      # compute using a local hash - involves zero IPC
      $lcount{ $ten[ rand @ten ] }++ for $seq->[0] .. $seq->[1];

      # increment shared hash - one IPC per key
      while ( my ( $key, $val ) = each %lcount ) {
         $count->incrby( $key, $val );
      }
   }
)->run;

printf "$_ -> %ld\n", $count->get($_) for @ten;
printf "duration: %0.3f seconds\n", time - $start;
[download]

Results running Perl v5.28, parallel time includes workers spawning-shutdown

  Serial 1.534 seconds
Parallel 0.410 seconds

0 -> 999455
1 -> 1000312
2 -> 999828
3 -> 1001949
4 -> 999227
5 -> 997375
6 -> 1001048
7 -> 999806
8 -> 1000212
9 -> 1000788
[download]

Regards, Mario

Comment on Re^2: Camel vs. Gopher Select or Download Code

Replies are listed 'Best First'.

Re^3: Camel vs. Gopher
by marioroy (Prior) on Dec 13, 2018 at 01:37 UTC

The pipeline method batches multiple operations into a single IPC call. A smaller chunk_size value is needed to notice the difference.

      # increment shared hash - single IPC per chunk
      $count->pipeline(
         map { [ 'incrby', $_ => $lcount{$_} ] } keys %lcount
      );
[download]

Comparison chunk_size => 1_000

Before 0.814 seconds  $count->incrby(...)
 After 0.522 seconds  $count->pipeline(...)
[download]

Regards, Mario

[reply]
[d/l]
[select]


XP is just a number
	PerlMonks