Just an update on how I got step one working -- piping standard error and standard out to their own files on a per-job basis (so each job gets a separate set of log files). The suggestions for just reopening STDERR and STDOUT weren't working because I was opening and then closing them inside a while loop, and when I closed them, and then later reopened, they were getting new fileno() (as a previous poster mentioned, I should check that the number is 2). Instead of closing them at the end of the loop, I just reopened them back to /dev/null and everything worked fine. Here is some psuedocode that may make it easier to understand:
while(my $job = get_next_job())
{
my $run_path = $job->get_run_path();
open STDERR, '>', "$run_path/stderr.txt" or die;
open STDOUT, '>', "$run_path/stdout.txt" or die;
$job->run();
open STDERR, '>', '/dev/null';
open STDOUT, '>', '/dev/null';
# Do some other job cleanup/notification stuff that may
# output to STDERR/STDOUT but I don't want in the
# individual job log files
do_some_other_stuff_before_going_to_next_job();
}
Thanks everyone for the help.