Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Efficient way to fork processes and keep a running count?

by bfreemer (Novice)
on Jan 27, 2010 at 19:27 UTC ( #820009=perlquestion: print w/replies, xml ) Need Help??

bfreemer has asked for the wisdom of the Perl Monks concerning the following question:

I'm a bit new to IPC and forking.

I'm trying to read a series of CDs (for example 12 discs in a collection). I have 3 CD drives. I will fork a child process for each drive. The child will output a file, e.g. "disc1.wav" and then exit.

How can I best have the parent keep track of the number of discs so far read and pass the appropriate value to each child in sequence?

The scenario I fear:

Parent -> child 1, cdrom1 still reading disc1 -> child 2, cdrom2 *finished* reading disc2 -> exit0 -> child 3, cdrom3 still reading disc3
Child 2 exits as above. Next, the parent should fork another read child:
Parent -> child 1, cdrom1 still reading disc1 -> child 2 (done) -> child 3, cdrom3 still reading disc3 -> child 4, cdrom2 read disc4
The parent should pass the disc number "4" do child 4 so it knows which disc its reading. Seems tricky! Any tips or tutorial links?

Replies are listed 'Best First'.
Re: Efficient way to fork processes and keep a running count?
by ikegami (Pope) on Jan 27, 2010 at 20:27 UTC
    use threads; use Thread::Queue qw( ); my $num_drives = 3; my $num_discs = 12; my $q = Thread::Queue->new(); $q->enqueue( 1..$num_discs ); $q->enqueue( (undef) x $num_drives ); for my $drive (1..$num_drives) { ... eject disc ... async { while (defined(my $disc = $q->dequeue())) { print("Please insert disc $disc in drive $drive\n"); ... wait for disc to be present ... ... read disc ... ... eject disc ... } }; } $_->join for threads->list();
      Thanks much ikegami for the quick reply and code example. I'll do some reading now and play with your code to learn a little. Thanks again for the example!
Re: Efficient way to fork processes and keep a running count?
by bart (Canon) on Jan 27, 2010 at 19:49 UTC
    fork isn't so bad for efficiency. So I suggest you let the parent do all the bookkeeping, and you wait for any of the 3 children to finish. And then you can start up the next disk in the available drive.
Re: Efficient way to fork processes and keep a running count?
by BioLion (Curate) on Jan 27, 2010 at 20:17 UTC

    I personally like Parallel::Forker - it makes polling processes easy, and also gives you a lot of control over them, so you can monitor them pretty closely. Have a look and see what you think.

    Just a something something...
      Thanks much for the reference! Off to do some reading...
Re: Efficient way to fork processes and keep a running count?
by JavaFan (Canon) on Jan 27, 2010 at 23:24 UTC
    There are several ways. The easiest is to assume reading each CD takes the same amount of time. Then each child reads 4 disks, and there's hardly any bookkeeping needed.

    Otherwise, you could implement a simple queue - each child fetches the next number from the queue, reads that disk, and repeats until the queue is empty. You could use a file or a database table for the queue (remember to lock!). Or you could have the children read from a pipe.

    Another idea is to have each child read one CD. If the CD has been read, the child terminates. Parent notices the child terminates (wait/waitpid) and forks a new one. New child should be passed both a drive and a CD number.

Re: Efficient way to fork processes and keep a running count?
by ikegami (Pope) on Jan 28, 2010 at 06:13 UTC
    A forking version:
    use POSIX qw( _exit ); my $num_drives = 3; my $num_discs = 12; my @idle; # Idle drives my %busy; # Busy drives, keyed by the pid using them sub wait_for_a_kid { my $pid = wait(); if ($pid == -1) { warn("Can't wait: $!\n"); return 0; } ... check the child's exit code if desired ... push @idle, delete($busy{$pid}); return 1; } for my $drive (1..$num_drives) { ... eject disc ... push @idle, $drive; } my @queue = 1..$num_discs; while (@queue) { if (!@idle) { last if !wait_for_a_kid(); } my $disc = shift(@queue); my $drive = shift(@idle); my $pid = fork(); if (!defined($pid)) { warn("Can't fork: $!\n"); last; } if ($pid) { $busy{$pid} = $drive; next; } if (!eval { print("Please insert disc $disc in drive $drive\n"); ... wait for disc to be present ... ... read disc ... ... eject disc ... 1; }) { warn($@); _exit(1); } _exit(0); } while (keys(%busy)) { last if !wait_for_a_kid(); }

    Update: Added some error checking. Fixed a bug exit wait loop.

Re: Efficient way to fork processes and keep a running count?
by targetsmart (Curate) on Jan 28, 2010 at 05:33 UTC

    As per the scenarios you showed, IMHO, there is no need to fork a fourth child, the second child can do the read of disc4 once it is done with disc2.
    As only 3 drives exist, 3 processes at a time would be sufficient, that too reusing the existing pool of processes without forking a new one is a good idea.
    Forking more number of processes would increase unnecessary complexity.


    Vivek
    -- 'I' am not the body, 'I' am the 'soul'/'consciousness', which has no beginning or no end, no attachment or no aversion, nothing to attain or lose.
      You are violently agreeing with the OP. He said the fourth child should be forked after one of the original three had exited, meaning at most three children exist at any given time.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://820009]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (3)
As of 2020-12-01 05:45 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?