learnedbyerror:

The last time I did a system like that it worked pretty much like this:

#!/usr/bin/perl
use strict;              
use warnings;            
                         
my %Jobs = (             
    Action1 => { status=>'READY', },                                  
+          
    Action1a=> { status=>'READY', prereqs => [ qw(Action1) ], },
    Action2 => { status=>'READY', },
    Action2a=> { status=>'READY', prereqs => [ qw(Action2) ], },
    Action3 => { status=>'READY', prereqs => [ qw(Action1a Action2a) ]
+, },
    Action4 => { status=>'READY', prereqs => [ qw(Action3) ], },
);                       
                         
my @task_list = sort keys %Jobs;
                         
my $cnt=0;               
while (@task_list) {     
    ++$cnt;              
    print "\nTIME: $cnt\n";
    for (sort keys %Jobs) {
        printf "%-16.16s %-10.10s %u\n",
            $_, $Jobs{$_}{status}, is_ready($_);
    }                    
                         
    # Clean out the items that are already done
    @task_list = grep { $Jobs{$_}{status} ne 'DONE' } @task_list;
                         
    # Each time we wake up, get list of tasks ready to go and fire
    # them off           
    my @ready = grep { is_ready($_) } @task_list;
    $Jobs{$_}{status}="RUNNING" for @ready;
                         
    # Fake code to pretend to monitor tasks & update job list
    for (@task_list) {   
        $Jobs{$_}{status}="DONE"
           if $Jobs{$_}{status} eq "RUNNING" and rand() < 0.25;
    }                    
    sleep 1;             
}                        
                         
sub is_ready {           
    my $task=shift;      
    return 0 if $Jobs{$task}{status} ne 'READY';
    return 1 if ! exists $Jobs{$task}{prereqs};
    for (@{$Jobs{$task}{prereqs}}) {
        return 0 if $Jobs{$_}{status} ne 'DONE';
    }                    
    return 1;            
}
[download]

Basically, each task knows what its prerequisite tasks are, and will fire off only when all its prerequisites are satisfied. The original also handled error cases by simply restarting the failed jobs after a delay. (On that system, most failures were due to missing files, so re-running was sufficient to fix most problems.) Also, I used a database as the scratchpad, so that the system could recover properly if the system went down and then recovered.

I'd post the code, but it was C# rather than perl.

...roboticus

When your only tool is a hammer, all problems look like your thumb.