Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re^2: IPC::Msg Fork queue

by marioroy (Prior)
on Feb 24, 2019 at 05:08 UTC ( [id://1230484]=note: print w/replies, xml ) Need Help??


in reply to Re: IPC::Msg Fork queue
in thread IPC::Msg Fork queue

Hi trippledubs,

MCE can process a sequence of numbers efficiently. E.g. 1 to MAX_ITEMS. The bounds_only option tells MCE to pass the next starting and ending elements only. Hence boundaries only. IPC occurs once per each chunk_size.

#!/usr/bin/env perl use strict; use warnings; use MCE; use constant MAX_ITEMS => 200000; my $n_workers = shift // do { die "usage: $0 n_workers\n"; }; sub isPrime { my $num = shift; return 1 if ($num < 4); return 0 if ($num %2 == 0); for (my $i=3; $i <= sqrt($num); $i+=2) { return 0 if $num % $i == 0; } return 1; } sub task { my ( $mce, $seq, $chunk_id ) = @_; for my $data ( $seq->[0] .. $seq->[1] ) { print "Prime: $data\n" if isPrime($data); } } MCE->new( max_workers => $n_workers, sequence => [ 1, MAX_ITEMS ], bounds_only => 1, chunk_size => 200, user_func => \&task, )->run();

Regards, Mario

Replies are listed 'Best First'.
Re^3: IPC::Msg Fork queue
by marioroy (Prior) on Feb 24, 2019 at 05:28 UTC

    Hi again,

    If needed, workers may write output orderly and efficiently via MCE::Relay simply by passing the init_relay option. The value to init_relay is not used in the demonstration. Just the block to MCE::relay which runs serially and orderly.

    #!/usr/bin/env perl use strict; use warnings; use MCE; use constant MAX_ITEMS => 200000; $| = 1; # necessary when workers output orderly my $n_workers = shift // do { die "usage: $0 n_workers\n"; }; sub isPrime { my $num = shift; return 1 if ($num < 4); return 0 if ($num %2 == 0); for (my $i=3; $i <= sqrt($num); $i+=2) { return 0 if $num % $i == 0; } return 1; } sub task { my ( $mce, $seq, $chunk_id ) = @_; my $output = ''; for my $data ( $seq->[0] .. $seq->[1] ) { $output .= "Prime: $data\n" if isPrime($data); } MCE::relay { print $output }; } MCE->new( max_workers => $n_workers, sequence => [ 1, MAX_ITEMS ], bounds_only => 1, chunk_size => 200, user_func => \&task, init_relay => 1, )->run();

    Regards, Mario

Re^3: IPC::Msg Fork queue
by trippledubs (Deacon) on Feb 26, 2019 at 15:32 UTC

    Love it! What I wanted to get across is multiple workers operating, enqueing and dequeing, inside same queue. Like Thread::Queue or MCE::Queue. Checking if a number is prime, probably not the best example, but easy to understand. That is not much different than what Parallel:ForkManager can do. Re: Sum to 100 at Rosetta Code is the better example. If you can get your problem into a queue, and keep possible solutions inside the queue, then that design can be made to be concurrent. The best example would probably be an Evolutionary Algorithm or something like that. Queues are awesome anyways, fun to say, rhyme with a lot of other words, lot of vowels.. Not communist and unfair like a stack.

    They are primitives that can be used as building blocks, observable, core perl, been around forever, reentrant, configurable, and persistent. Seems to scale well and does priorities too. And with priorities comes the ability to order the output and sort. API stinks, but documented well. From what I read just a linked list inside the kernel. But it doesn't do chunking for you like MCE, well it doesn't do a lot of things MCE does for you, which as you illustrate is a big bottleneck.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1230484]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others wandering the Monastery: (7)
As of 2024-03-28 10:35 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found