Concurrent Processes

mcogan1966 has asked for the wisdom of the Perl Monks concerning the following question:

Consider the following bit of code:

for ($i=0; $i<$g; $i++) {
  pipe ($rh, $wh)          # both of these are generated in the loop
  if ($pid[$i]=fork()) {   # parent process
    waitpid($pid[$i], 0);  # wait for child
    close ($wh);
    while (<$rh>) {        # gimme the output
      if ($_ =~ m/^Error/) { push (@error, $_); }
      elsif ($_ =~ m^Hits/) { push (@hits, $_); }
      else { push (@data, $_); }  # each is a result from an engine
    }
  }
  else {
    close ($rh);
    open (STDOUT, ">&$wh");
    open (OUT, "| $caller");  # call the handler with arguments
    close (OUT);
    exit(0);
  }
}
[download]

This is part of a multi-headed search engine. The process is to go out to several search engines, poll them for results, and pass them back to the main program. The program is customized to allow from 10-50 results from each search engine. If I say get 10 or 20 from each engine, it works fine. But if I go for 30 or more, it chokes. Is it possible that I'm trying to pull too much through the pipe?

Yes, I know there is stuff missing, but nothing that is relevant to the question at hand. All of that I can't share here.

I should also point out that the idea is to get all the searches to run at the same time so hopefully the entire process will speed up.

Comment on Concurrent Processes Download Code

Replies are listed 'Best First'.
Re: Concurrent Processes by ptkdb (Monk) on Oct 21, 2003 at 18:11 UTC
What do you mean by 'chokes', hangs, segfaults, etc? And some of the stuff(lots of it apparently, including what 'caller' happens to be) may be relevant to why it's 'choking'. Also this bit I think is incorrect on its face: `if ($pid[$i]=fork()) { # parent process waitpid($pid[$i], 0); # wait for child close ($wh); ...` [download] After each fork, the parent process is going to wait for the most recently forked child to complete BEFORE the next one is forked. All of your forked processes are going to run SERIALLY not in parallel. What I think you might want to try is this: `## ## launch children ## for( $i = 0 ; $i < NUMPROCS ; $i++ ) { $pid[$i] = fork() ; next if( $pid[$i] != 0 ) ; # fork next process ## ## child stuff ## exit(0) ; # when finished } ## ## gather children ## foreach( @pid ) { waitpid($_, 0) ; # waits for each child if( $? != SOMETHINGGOOD ) { # bad exit status } }` [download]	[reply] [d/l] [select]
Re: Re: Concurrent Processes by mcogan1966 (Monk) on Oct 21, 2003 at 18:32 UTC
I see what you are saying. Acutally, I think the first part of what you said may be the cause of the problem, I'm still ending up with this running in SERIAL with the waitpid() inside the for loop. But I'm unclear as to how your code can be used to do what I need. The parent needs to be able to see what the child puts out. Since I'm no longer doing the 2-way pipe that was set up, where does the child give it's data back to the parent process? You threw in comments about 'child stuff' and 'gather children', but your design changes the way it all works. Could you be more clear about that?	[reply]
Re: Re: Re: Concurrent Processes by mpeppler (Vicar) on Oct 21, 2003 at 18:37 UTC
I think you should look at the select() call to check for output from your various children. May I recommend going to a bookstore and purchasing Stevens' Unix Network Programming? This will have a lot of examples about how to do this sort of thing, coded in very clear C. Michael	[reply]
Re: Re: Re: Re: Concurrent Processes by mcogan1966 (Monk) on Oct 21, 2003 at 18:48 UTC
Re: Re: Re: Concurrent Processes by ptkdb (Monk) on Oct 21, 2003 at 18:35 UTC
That bit of code was only addressing the launching and gathering of child processes. I can't/won't/shouldn't work out everything for you.	[reply]
Re: Re: Re: Re: Concurrent Processes by ptkdb (Monk) on Oct 22, 2003 at 17:03 UTC
Re: Concurrent Processes by QM (Parson) on Oct 21, 2003 at 23:18 UTC
for ($i=0; $i<$g; $i++) Idiomatically, this is foreach my $i (0..$g) Later on you have pipe ($rh, $wh) ...but then in the parent... close ($wh); ...or in the child... close ($rh); Why do you need a 2-way pipe just to close one of them down? Here's a generic form of parallel process dispatcher that I worked up a few years ago. I'm sure there some things that could be done better. I've reworked it to be generic, but I can't completely test it here (the "-\|" fails where I am now). Read more... (6 kB) -QM -- Quantum Mechanic	[reply] [d/l]
Re: Concurrent Processes by Skeeve (Parson) on Oct 22, 2003 at 08:15 UTC
There is another solution to your problem in your other thread.	[reply]


go ahead... be a heretic
	PerlMonks