http://qs321.pair.com?node_id=585853

chanakya has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

I'm trying to write a program which will calls a subroutine with a given sourceid.
There can be around 200 sourceids which are nothing but directory names.
The script should initially spawn 4 childs and call the subroutine with a
given sourceid (the sourceid cannot be repeated). Script should create new childs
as and when a child finishes the job and continue until all the 200 sourceids are processed.

Below is the code I've tried. My only question is
Is there anyway of doing the same thing in a simple manner and the most important thing
how do I pass the sourceids(unique) to the subroutine.
#!/usr/bin/perl -w
my $NumChilds = 4;

@sourceids=("aex", "athena", "athens","bidapa","bbic","bony","capctha"
+,"cooldo","italy","paris","xmltech");
my $sourcecount =  scalar(@sourceids)."\n";



# This section spawns the initial 4 children.

for ($counter = 1; $counter <= $NumChilds; $counter++) {

#foreach my $source (@sourceids){

        $pid = fork();

        if ($pid) # parent
        {
        $child{$pid} = $counter;
        print "forked process $counter.\n";
        }

        else # child
        {
        callScan();
        #should be callScan($source);
        exit $counter;
        }
#  }
}


#  spawns additional children .

$diecount = 0;

while ($counter <= $sourcecount)
{
        $doneproc = wait();
        $doneval = $? >> 8;
        $pid = fork();

        if ($pid) # parent
        {
        $child{$pid} = $counter;
        print "child $doneval ($doneproc) exited, forking process $cou
+nter.\n";
        $counter++;
        $diecount++;
        }

        else # child
        {
        callScan();
        #should be callScan($source);
        exit $counter;
        }
}


#  waits for all children to die.

while ($diecount <= $sourcecount)
{
        wait();
        $diecount++;
}

# subroutine.
sub callScan
{
        sleep(2);
        print "callScan with counter = $counter.\n";
}
print "Done.\n";

[download]
Thanks in advance

Replies are listed 'Best First'.
Re: Fork limited childs and passing parameters
by zentara (Archbishop) on Nov 24, 2006 at 13:41 UTC
    Check out Parallel::ForkManager such as controlling child processes

    Or do it manually with these old beauties: 

    #!/usr/bin/perl

=head1
#by Abigail of perlmonks.org
#Some times you have a need to fork of several children, but you want 
+to
#limit the maximum number of children that are alive at one time. Here
#are two little subroutines that might help you, mfork and afork. They
+ are very similar.
#They take three arguments,
#and differ in the first argument. For mfork, the first
#argument is a number, indicating how many children should be forked. 
+For
#afork, the first argument is an array - a child will be
#forked for each array element. The second argument indicates the maxi
+mum
#number of children that may be alive at one time. The third argument 
+is a
#code reference; this is the code that will be executed by the child. 
+One
#argument will be given to this code fragment; for mfork it will be an
+ increasing number,
#starting at one. Each next child gets the next number. For afork, the
+ array element is
#passed. Note that this code will assume no other children will be spa
+wned,
#and that $SIG {CHLD} hasn't been set to IGNORE.
=cut

#mfork example
mfork (10,10,\&hello);
sub hello{print "hello world\n";}
print "all done now\n";


#afork example
if ($#ARGV < 0){@ARGV = qw( 1 2 3 4 5)}
afork (\@ARGV,10,\&hello);
print "Main says: All done now\n";

sub hello{
my $data = $_[0];
print "hello world from $data\n";}

###################################################
sub mfork ($$&) {
my ($count, $max, $code) = @_;
foreach my $c (1 .. $count) {
wait unless $c <= $max;
die "Fork failed: $!\n" unless defined (my $pid = fork);
exit $code -> ($c) unless $pid;
}
1 until -1 == wait;
}
##################################################
sub afork (\@$&) {
my ($data, $max, $code) = @_;
my $c = 0;
foreach my $data (@$data) {
wait unless ++ $c <= $max;
die "Fork failed: $!\n" unless defined (my $pid = fork);
exit $code -> ($data) unless $pid;
}
1 until -1 == wait;
}
#####################################################

    [download]
    I'm not really a human, but I play one on earth. Cogito ergo sum a bum
[reply]
[d/l]
Re: Fork limited childs and passing parameters
by themage (Friar) on Nov 24, 2006 at 12:40 UTC
[reply]
Re: Fork limited childs and passing parameters
by salva (Canon) on Nov 24, 2006 at 14:57 UTC 
    use Proc::Queue size => 4, qw(run_back waitpids);

my @pids;
for my $source (@sourceids) {
  push @pids, run_back {
    callScan($source);
  };
}

waitpids(@pids);

    [download]
[reply]
[d/l]
Re: Fork limited childs and passing parameters
by chanakya (Friar) on Nov 24, 2006 at 14:06 UTC
    themage,zentara thank you for your suggestions and comments.
    zentara the mfork and afork are real gems thank you for sharing.

    Thank you
[reply]
Re: Fork limited childs and passing parameters
by chanakya (Friar) on Nov 27, 2006 at 12:43 UTC
    salva thanks for the suggestion of Proc::Queue. Its simple and very effective. zentara I'm interested to know about afork() routine.
    Please walk me through the code. I want to know how the childs are reaped by the parent and how the passed code is executed by the afork() routine

    sub afork (\@$&) {
my ($data, $max, $code) = @_;
my $c = 0;
foreach my $data (@$data) {
wait unless ++ $c <= $max;
die "Fork failed: $!\n" unless defined (my $pid = fork);
exit $code -> ($data) unless $pid;
}
1 until -1 == wait;
}

    [download]
[reply]
[d/l]

Back to Seekers of Perl Wisdom