http://qs321.pair.com?node_id=176222

moof1138 has asked for the wisdom of the Perl Monks concerning the following question:

I have a program that I would like to fire off some number of child processes (each a perl script with a separate argument that the parent supplies) to do a somewhat time consuming thing. Right now I want it to fire off 30 sub-processes, at some point in the future that number will likely grow to about 150. I would like all the child processes to be started at about the same time, since the thing they are doing - building reports on different fairly busy and occasionally slow to respond servers, and putting those reports in a database - is somewhat time consuming but not bandwidth or CPU consuming. It would be fine if it died after the children were fired off. The script currently serially builds reports and it is slow and can be unreliable if a remote host decides to quit cooperating or kill its session, which is why I am thinking this approach would be great. As I understand them, system() and backticks wait for the process to finish (even if you use &), while fork(), and exec() seem to only replace a single running process with another. Is what I want to do possible with Perl? If so how?

Replies are listed 'Best First'.
Re: spawning multiple child processes
by pjf (Curate) on Jun 21, 2002 at 05:05 UTC
    Sounds like a job for the Perl cook-book to me. Section 17.11 (Forking Servers) has an example of a server that forks off a new process for each connection it receives. This is similar to what you're doing, except I presume you wish to fork off a new process for each server in your list.

    The fork call creates a duplicate of a process, called a child. This child can do whatever it likes, independantly to the parent (including using exec to replace itself with another program).

    A very basic framework for what you're after might look like this (adapted from the code mentioned above in the Perl Cookbook):

    #!/usr/bin/perl -w use strict; my @servers = qw(LIST OF SERVERS); foreach my $server (@servers) { my $pid; next if $pid = fork; # Parent goes to next server. die "fork failed: $!" unless defined $pid; # From here on, we're in the child. Do whatever the # child has to do... The server we want to deal # with is in $server. exit; # Ends the child process. } # The following waits until all child processes have # finished, before allowing the parent to die. 1 while (wait() != -1); print "All done!\n";

    That should hopefully give you a good start on solving your problem. Don't hestiate to ask questions if anything doesn't make sense.

    All the best,

    Paul Fenwick
    Perl Training Australia

Re: spawning multiple child processes
by kvale (Monsignor) on Jun 21, 2002 at 05:11 UTC
    You can definitely create child processes in Perl programs. The most common way is to do this with fork:
    sub bear_child { my $pid; my $sigset; # block signal for fork $sigset = POSIX::SigSet->new(SIGINT); sigprocmask( SIG_BLOCK, $sigset) or die "Can't block SIGINT for fork: $!\n"; # fork a subprocess die "fork: $!" unless defined ($pid = fork); if ($pid) { # Parent records the child's birth and returns. sigprocmask( SIG_UNBLOCK, $sigset) or die "Can't unblock SIGINT for fork: $!\n"; $children{$pid} = 1; $children++; return; } else { # Child process must *not* return from this subroutine. $SIG{INT} = 'DEFAULT'; # make SIGINT kill us as it did befo +re # unblock signals sigprocmask( SIG_UNBLOCK, $sigset) or die "Can't unblock SIGINT for fork: $!\n"; # do kid sutff # ... print "child $$ doing stuff\n"; # cleanup print "child $$ is done processing\n"; exit; # make sure sub never returns } }
    So here, the fork makes a clone of the parent process; both processes then test the $pid and take different branches according to whether they are the parent or the child.

    -Mark
•Re: spawning multiple child processes
by merlyn (Sage) on Jun 21, 2002 at 13:34 UTC
      I would like to propose a new acronym that could save everyone here a lot of typing:
      CMFC
      Check Merlyn's friendly colunms / check my friendly columns (if you are Merlyn)
      ()-()
       \"/
        `                                                     
      
Re: spawning multiple child processes
by DamnDirtyApe (Curate) on Jun 21, 2002 at 06:44 UTC
      sigh...no article at that url...that thing about posting something on the internet makes it live forever? apparently not true for actually useful stuff. Will execute the Googleit routine.
Re: spawning multiple child processes
by Abigail-II (Bishop) on Jun 21, 2002 at 09:11 UTC
    Well, exec replaces a single process with another, but fork copies the running process. So, fork combined with exec is the way (at least under Unix) to spawn off new tasks. If you are doing a system, behind the screens, Perl will do a fork and an exec. It will just wait for the child to finish before continuing. Using a system with an ampersand tacked to the end of the command is an option, but that means Perl will first invoke a shell (using fork and exec) which will spawn off your command (using another fork and exec).

    Abigail

Re: spawning multiple child processes
by jepri (Parson) on Jun 21, 2002 at 05:17 UTC
    You are looking for the fork command. Fork creates two running processes, one the original and one the clone. A process may fork itself as often as it wants. This is how you do your 150 child processes.

    Several people have built modules around fork, such as parallel::forkmanager or parallel::taskmanager can't remember names.

    In your case, however, I think you just want to fork 150 times. Read the man page very closely, it takes a bit to wrap your mind around it.

    ____________________
    Jeremy
    I didn't believe in evil until I dated it.

Re: spawning multiple child processes
by moof1138 (Curate) on Jun 21, 2002 at 04:50 UTC
    Hmm. I was thinking about this, and I just though maybe, I could use system(), use POSIX: setsid with the children to get them to separate (and set their STDIN and STDOUT to /dev/null, etc.), and have the system() call finish off that way. I think that would work. Naturally I think of it a minute after I press submit...
Re: spawning multiple child processes
by moof1138 (Curate) on Jun 21, 2002 at 14:22 UTC
    Thanks to all for the pointers, you have all been immensely helpful. Mr DamnDirtyApe, I read the Mr. Peabody article which helped make more clear the perdoc page I had been stumbling through, many thanks. And merlyn, thanks for your article, it seems to be a perfect starting point for doing what I am now planning - BTW it looks like some individuals are downmodding you, which is weird since your column did a pefect job of going over what I was really looking for.

    BTW - I actually misdescribed why I had issues with fork(), since I realized that it spawned off a separate copy of the parent, I just thought at the time that that was not what I wanted, since I was thinking in terms of two separate scripts. So, what I had originally wanted to do (it's all so obvious now) is a fork() and then an exec() of the second script. Of course, I do not think that is necessary now that I see better how it works, since I had not really 'gotten' that data structures allocated by the parent are copied in the child, so there is no real need to exec() a separate script just so I could send it some args.