Starting with a script does around a dozen lengthy analysis tasks, so I'm trying to update it to use a cluster and do the jobs in parallel. Each analysis script is submitted with a load sharing facility (LSF) Using LSF to monitor the analysis jobs is frowned upon, as that tends to put a load on LSF and keep it from doing more important things. So, I need to use flag files that tell me when each analysis is done.

This submits a job and waits for completion:

use strict;
my $jobID = `bsub -q long -o $cl_log -J seqPipe "$jobCmd"`;
print "Waiting for analysis work.\n";

while (1){ 
    last if ( -e "$resultsDir/analysis_completed" );
    print LOG ".";
    sleep 15;
}

print "\nFound completion flag.  Analysis jobs are done.\n";
[download]

Assuming each analysis job creates its own flag to indicate completion, this will get more complicated when I'm trying to monitor a dozen jobs. Later on, there's the possibliity that some jobs will have their own set of analysis jobs that they need to monitor.

I'm thinking this while statement will end up with a set of tests, one for each job. Still, it seems inefficient to keep testing for all those files until the last one has finished. Any ideas on better ways to grow this system?