Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Waiting For Multiple Programs to Complete

by NathanE (Beadle)
on Oct 07, 2005 at 21:28 UTC ( [id://498321]=perlquestion: print w/replies, xml ) Need Help??

NathanE has asked for the wisdom of the Perl Monks concerning the following question:

Hello everyone;

I recently divided a large perl program up into smaller scripts to attempt to take advantage of multiple processors on a system. The model is that I'd have a central control program that launches each of the seperate programs, which work parallel to each other, takes the results from those programs (result files and the fact that they're complete) and decides on how best to take advantage of multiple processes. It does this by knowing how many requests are waiting to be executed at various stages of the process.

Now - my problem is, how can I run two programs simultaneously, wait for the results of one and make the next choice occording to the new numbers, while the other is still running and so on and so forth. I can make it wait on a single process to complete, but not sure how to do multiple.

Thanks Monks.

UPDATE: Solution is to use POE to handle the multiple events - thanks for the solution fauria
  • Comment on Waiting For Multiple Programs to Complete

Replies are listed 'Best First'.
Re: Waiting For Multiple Programs to Complete
by fauria (Deacon) on Oct 07, 2005 at 22:14 UTC
    This sounds a lot like event driven programming. I would think about POE for this.
      While POE is the benchmark for many when it comes to event-based programming in Perl, it should be noted that there are other module alternatives which facilitate this programming style - In particular, I have been using the Event module quite extensively of late and found it to be extremely flexible and direct in its operations. The situation where I have been using it too (RTSP delivery for digital video-on-demand), using POE with all of its dependencies and its operational overhead, may have proven problematic. If you do decide to take a look at Event, make sure you also take a look at the tutorial PDF file in the distribution in addition to the module POD.

       

      perl -le "print unpack'N', pack'B32', '00000000000000000000001000000000'"

      That's a PERFECT idea! Thanks!
Re: Waiting For Multiple Programs to Complete
by Zaxo (Archbishop) on Oct 07, 2005 at 21:38 UTC

    The wait function returns the pid of a completed child process. If you keep a hash with pid as key and whatever identifying data you like as value, you can tell which subtask has finished.

    After Compline,
    Zaxo

      The problem with this is that I can only run one program at a time (as wait literally waits before doing anything else) when I need to complete parallel tasks and react to task completion etc. I don't think wait can do this, but I may be wrong?

      My other option is to create an independant hash table and provide each program a unique ID that it uses to store its status in the hash table. Then, I can just cycle over this hash key waiting for a change of status to what I'm looking for and continuing.. BUT - I'd rather find a more native way to do it..

      Also, if I were to do it that way, does anyone know the most efficient way to keep checking for this hash change? I don't want to loop over it constantly as that would probably consume too much CPU.. Maybe set a sleep time within a loop - anyone have experience with anything along these lines?

      Thanks again everyone.

        The wait call is a form of sleep. It sleeps until SIGCHLD if there are no completed children queued up for attention. If there are, it returns the pid of the one it serviced.

        Your application might look like this skeleton,

        our %kid; sub forkus { my $task = shift; # it's a hash ref defined(my $cpid = fork) or warn $! and return; $kid{$cpid} = $_->id, return $cpid if $cpid; exec $_->cmd; exit -1; } forkus($_) for (@first_tasks); while (%kid) { my $cpid = wait; # sleep until somebody's done forkus(next_task($cpid)) if more($cpid); # suitably defined delete $kid{$cpid}; }
        I've glossed over the design of the task hashes and the next_task() and more() functions. They will be application dependent.

        POE may be just what you need, but I find it somewhat difficult to work with. Lack of practice, perhaps.

        After Compline,
        Zaxo

        Try:
        use POSIX ":sys_wait_h"; waitpid(-1,WNOHANG);
        to check for any waiting processes, and return immediately instead of blocking if there aren't any.

        Or, use a handler for $SIG{CHLD} to do your thing, and handle processes exiting as they come in.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://498321]
Approved by sauoq
Front-paged by monkfan
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (7)
As of 2024-04-19 08:28 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found