Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Re: Help with multiple forks

by Eliya (Vicar)
on May 30, 2012 at 16:37 UTC ( [id://973336]=note: print w/replies, xml ) Need Help??


in reply to Help with multiple forks

Getting back to Parallel::ForkManager, which you attempted to use in your first post, you might want to try

use Parallel::ForkManager; my $fm1 = new Parallel::ForkManager(5); for my $val1 (1..10) { $fm1->start and next; # $0 = "processing step 1: $val1"; sleep 10; open FILE, ">", "$val1.txt" or die $!; print FILE "Output of step 1\n"; close FILE; my $fm2 = new Parallel::ForkManager(2); for my $val2 (qw/a b/) { $fm2->start and next; # $0 = "processing step 2: $val1/$val2"; sleep 5; open FILE1, "<", "$val1.txt" or die $!; open FILE2, ">", "$val2$val1.txt" or die $!; while (my $line = <FILE1>) { $line =~ s/1/2/; print FILE2 $line . "\n"; } close FILE1; close FILE2; $fm2->finish; } $fm1->finish; }

This would run max 5 processes at the outer level, plus max 2*5 at the inner level, i.e. max 15 processes total (10 inner from the last round + 5 outer for the next round).  I'm not entirely sure that's what you want, but it's at least something to play with...

P.S. If you uncomment the $0 = ... lines, you can grep for "processing" in the ps output to observe what's going on.

Replies are listed 'Best First'.
Re^2: Help with multiple forks
by Anonymous Monk on May 31, 2012 at 09:33 UTC

    Thanks a lot! That's exactly what I was trying to write. I'm a little ashamed because it seems I should have been able to come up with it myself.

    There is one funny thing, though: If I comment out the "for my $val2 (@array2) {..." and the corresponding closing curly bracket (and change $val2 to a constant like "step2"), it breaks with Cannot start another process while you are in the child process at .../perl/lib/perl5/Parallel/ForkManager.pm line 463. Not that I would want that - in my case I need the two nested loops. It's just that I don't know what is happening there. I mean, in the nested loop, I would have assumed I was in the 1st child process, which is itself the parent of the 2nd child process, right?

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://973336]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others scrutinizing the Monastery: (6)
As of 2024-03-28 12:42 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found