Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re: Quicker array processing

by kvale (Monsignor)
on Jun 27, 2002 at 17:27 UTC ( [id://177785]=note: print w/replies, xml ) Need Help??


in reply to fork() and batch processing mail (was: Quicker array processing)

It sounds like you are on the right track. I'd suggest the pre-forked server design as detailed in the Perl Cookbook. Basically, you fork off a process for each mail address until you have hit your desired max number of processes. Then wait for each child to finish and start another process at that time. Repeat until done. That will get the task done as quickly as possible, at some expense in space.

From another program I wrote:
my %children = (); # keys are current child process IDs my $children = 0; # current number of children # Install signal handlers for children. $SIG{CHLD} = \&REAPER; $SIG{INT} = \&HUNTSMAN; # create a set of children for (1..$max_connections) { my $group = shift @groups; bear_child( $group) if defined $group; last if @groups == 0; } # And maintain the population. MAINTAIN: while (@groups > 0) { sleep; # wait for a signal (i.e., child's death) for (my $i = $children; $i < $max_connections; $i++) { my $group = shift @groups; bear_child( $group); # top up the child pool last MAINTAIN if @groups == 0; } } # wait for the last of the children to complete scanning sleep while $children > 0; # reap any wayward children and exit cleanly HUNTSMAN(); sub REAPER { $SIG{CHLD} = \&REAPER; my $pid = wait; $children--; delete $children{$pid}; print "reaping child $pid\n"; } # handler for SIGINT and normal exit - kill all the children sub HUNTSMAN { local($SIG{CHLD}) = \&REAPER; # we're going to kill our children kill 'INT' => keys %children; exit; } # spawn subprocesses to search groups sub bear_child { my $group = shift; my $pid; my $sigset; # block signal for fork $sigset = POSIX::SigSet->new(SIGINT); sigprocmask( SIG_BLOCK, $sigset) or die "Can't block SIGINT for fork: $!\n"; # fork a subprocess die "fork: $!" unless defined ($pid = fork); if ($pid) { # Parent records the child's birth and returns. sigprocmask( SIG_UNBLOCK, $sigset) or die "Can't unblock SIGINT for fork: $!\n"; $children{$pid} = 1; $children++; return; } else { # Child process must *not* return from this subroutine. $SIG{INT} = 'DEFAULT'; # make SIGINT kill us as it did befo +re # unblock signals sigprocmask( SIG_UNBLOCK, $sigset) or die "Can't unblock SIGINT for fork: $!\n"; # open a connection to the news server undef $nntp; until ($nntp) { $nntp = Net::NNTP->new( $news_server, Timeout => 60); } $nntp->authinfo( $username, $password) if defined $username; print "child $$ searching $group\n"; # do assigned group process_group( $group); # close connection to news server $nntp->quit; print "child $$ is done processing $group\n"; exit; # make sure sub never returns } }
Replace @groups wth @address, and the NNTP stuff with your own processing. Season to taste.

-Mark

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://177785]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others examining the Monastery: (7)
As of 2024-04-18 15:54 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found