http://qs321.pair.com?node_id=237089


in reply to Run N similar tasks in parallel

Here's the pattern I used for that on a recent project -- it's a bit more robust than yours. 
use POSIX ":sys_wait_h";

my @tasks = (1..398);

my %kids;

{
  while (@tasks and keys %kids < 5) {
    $kids{fork_a_task(shift @tasks)} = "active";
  }
  {
    my $pid = waitpid(-1, 0);
    if ($pid == -1) {
      %kids = ();
    } else {
      delete $kids{$pid};
    }
  }
  redo if @tasks or %kids;
}

sub fork_a_task {
  my $i = shift;
  my $pid = fork;
  return $pid if $pid;
  unless (defined $pid) {
    warn "cannot fork: $!";
    return 0;
  }
  ## do stuff for task $i goes here...
  exit 0;
}

[download]

-- Randal L. Schwartz, Perl hacker
Be sure to read my standard disclaimer if this is a reply.

Replies are listed 'Best First'.
Re: &bull;Re: Run N similar tasks in parallel
by bronto (Priest) on Feb 20, 2003 at 14:47 UTC

    ++merlyn for checking fork's return value, that I forgot.

    Please, can you explain why your approach is more robust? I am really interested in it!

    Ciao and thanks!
    --bronto

    The very nature of Perl to be like natural language--inconsistant and full of dwim and special cases--makes it impossible to know it all without simply memorizing the documentation (which is not complete or totally correct anyway).
    --John M. Dlugosz
[reply]
[d/l]
      By checking the possible failure of fork, I never "think" I'm working on five kids when only two have really started.

      By checking the value of waitpid for a possible -1 return, I never "think" I have kids left when I really don't. The -1 force-resets the hash.

      -- Randal L. Schwartz, Perl hacker
      Be sure to read my standard disclaimer if this is a reply.

[reply]
        By checking the value of waitpid for a possible -1 return, I never "think" I have kids left when I really don't. The -1 force-resets the hash.

        Good, I should check for the return value of wait as well. Thanks.

        Sorry if I am bothering you about this, but I am missing an information: why do you prefer waitpid to wait? Flexybility? Possibility to make waitpid a non-blocking call? What?

        Thanks in advance.

        --bronto

        The very nature of Perl to be like natural language--inconsistant and full of dwim and special cases--makes it impossible to know it all without simply memorizing the documentation (which is not complete or totally correct anyway).
        --John M. Dlugosz
[reply]
[d/l]
[select]

In Section Cool Uses for Perl