Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

Re^2: Parallelization of multiple nested loops

by marioroy (Prior)
on Apr 29, 2019 at 00:09 UTC ( [id://1233096]=note: print w/replies, xml ) Need Help??


in reply to Re: Parallelization of multiple nested loops
in thread Parallelization of multiple nested loops

Hi, cmk

Very cool! Seeing my old parallel code made me want to try again for a 100% pure Perl solution not involving Inline::C. So first serial and parallel code afterwards. Workers consume about 16 MB each. The parallel code runs 3 times faster compared to serial. This is made possible by having workers write to STDOUT directly.

$ time perl serial.pl >/dev/null $ time perl parallel.pl >/dev/null

Serial Demonstration

use strict; use warnings; my @vals = qw( 0.0 0.2 0.4 0.6 0.8 1.0 ); sub proc { for my $a ( @vals ) { for my $b ( @vals ) { for my $c ( @vals ) { for my $d ( @vals ) { for my $e ( @vals ) { for my $f ( @vals ) { for my $g ( @vals ) { for my $h ( @vals ) { for my $i ( @vals ) { for my $j ( @vals ) { for my $k ( @vals ) { print "$a\t$b\t$c\t$d\t$e\t$f\t$g\t$h\t$i\t$j\t$k\t1\t1\n"; }}}}}}}}}}} } proc();

Parallel Demonstration

use strict; use warnings; use MCE; my @vals = qw( 0.0 0.2 0.4 0.6 0.8 1.0 ); # Must autoflush because workers write to STDOUT directly. STDOUT->autoflush(1); sub proc { my $mce = MCE->new( max_workers => scalar(@vals), chunk_size => 1, init_relay => 1, user_func => sub { my ($a, $b, $c) = @{ MCE->user_args }; my ($buf, $d ) = ( '', $_ ); # $d is the input for my $e ( @vals ) { for my $f ( @vals ) { for my $g ( @vals ) { for my $h ( @vals ) { for my $i ( @vals ) { for my $j ( @vals ) { for my $k ( @vals ) { $buf .= "$a\t$b\t$c\t$d\t$e\t$f\t$g\t$h\t$i\t$j\t$k\t1\t1\n"; }}}}}}} # Relay is driven by the chunk_id value behind the scene. # The benefit is orderly output, one worker at a time. MCE::relay { print $buf }; } )->spawn; for my $a ( @vals ) { for my $b ( @vals ) { for my $c ( @vals ) { # MCE workers persist between each run. The user_args option # is how to pass parameters to them. $mce->process({ input_data => \@vals, user_args => [ $a, $b, $c ], }); } } } $mce->shutdown; } proc();

Parallel happens at the 4th level to minimize memory consumption.

Regards, Mario

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1233096]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others chilling in the Monastery: (5)
As of 2024-03-29 02:36 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found