A syslog server is a very good (and, IME, very underused) solution.

Alternately, if that's not sexy enough to get management buy-in, you could instead set the processes up to all log to a central database, but that would mostly be just pointless overhead unless you're using a database already (and may still be pointless overhead even if you are).

++ Much better than my idea below, but there would need to be a reliable way to identify the cases where any of the 150 processes fail before they get to the point of sending their UDP packet to the log server. Not hard to handle, just easy to forget...

update: On second thought, if the log data from each host is anything more than a single summary report printed at the end of each job, I would still kinda prefer my approach. If the jobs are printing progress reports at intervals, the entries submitted to a central syslog server will tend to be interleaved, and will need to be sorted out. Not a big deal, obviously, but it might be handier to have the stuff "pre-sorted" by harvesting from each machine.

there would need to be a reliable way to identify the cases where any of the 150 processes fail

A 'job started' message could be sent by a wrapper which cares about the process and reports its exit status.

the entries submitted to a central syslog server will tend to be interleaved

syslog is configurable, and one could send the the log messages to different files based on level/facility and host. Anyways, the log line is marked with the host sending the log message, so sorting things out is as easy as grepping the log file for a host.

--shmem

_($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                              /\_¯/(q    /
----------------------------  \__(m.====·.(_("always off the crowd"))."·
");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}

It is interesting, the different backgrounds from which we all come. I'm predominantly used to applying solutions to assets given, while other people come from backgrounds where adding a server here or there is considered trivial.

It's good to get both perspectives.

I recommend these options:

• P.O.E. - invoke remotely with clear responses
• Local net delivery - cron response is sent to central machine via scp/rsync
• NFS - copy response to central common directory space
• MTA - send responses to a common mail address
• ... limited by your imagination and resources

In effect, the perl script does mechanically what you are now doing manually (you can be replaced by a perl script ;). Of course, that assumes that the 150 machines are all doing the same thing, and their results are all stored using the same path/file.name on each machine. Either that, or else the list of machines to scan includes all the details needed to find the log file for each one.

For doing the harvest, there's nothing inherently wrong about just running an appropriate command in back-ticks like my $log = `ssh $host cat /log/path/my_process.log` (this assumes you have public-key authentication in place, so the userid running this won't need to supply a password for each connection). If the overhead of launching a shell 150 times bothers you, you could do it like this:

use strict;
use POSIX;

my $today = strftime( "%Y%m%d", localtime );
mkdir "/path/harvest/$today";
chdir "/path/harvest/$today" or die $!;

my @remote_machine_list = ...  # (fill in the ...)

my $shell_pid = open my $shell, "|-", "/bin/sh" or die $!;

print $shell "cd /path/harvest/$today\n";

print $shell "ssh $_ cat /log/path/my_process.log > $_.log 2> harvest.
+errlog ||".
             " echo $_ failed 2> harvest.errlog\n"
    for ( @remote_machine_list );

print $shell "exit\n";
waitpid $shell_pid, 0;
[download]

You might need to add stuff to that, like setting SIG_ALARM in case ssh hangs on a given host. On each iteration, if the "ssh $_ ..." works, its output is stored locally in "$_.log", and any stderr output is appended to the local file "harvest.errlog". But if the ssh fails, a line about that is appended to "harvest.errlog" as well.

But you also have a variety of CPAN modules in the Net::SSH domain that you might find preferable.

This may work, I do have public-key authentication in place already. Thanks for the information. I will let you know what I end up doing.


After all this is over, all that will really have mattered is how we treated each other.

 ...
 # $addr is the actual subnet, e.g. "192.168.1.0"
 ...
 my $output = qx{nmap -sP ${addr}/24};
 my @nodes= $output =~ /(?<=\s)c\w+\b/g;
[download]

 ... 
 print header('text/html');
 print h1('Local Network: '. $addr . '/24');
 print map "$_ appears to be up<br />", @hosts
 ...
[download]

 ...
 my ($exe, $cmd) = ('/usr/bin/rsh', 'ps -fl r -u username');
 my $cnt = 0;

 for my $node ( sort @nodes ) {
   my @res = grep !(/$cmd/ || /STIME/), split /[\n\r]+/, qx{$exe $node
+ $cmd};
   my $nproc = scalar @res;   # how many processes

   if( $nproc ) {
      print map
         "Do " . "some ". "formatting of " . "ps -fl output here!",
           @res
   }
  ...
  ++$cnt
  ...
[download]

Hi perlfan,

The master daemon idea is somewhat attractive because I could apply this same technique to several other projects. I would be delighted to hear more details.

Best Regards


After all this is over, all that will really have mattered is how we treated each other.

"I have to log into each server via ssh and grep each log to see if the process completed"

oreilly.com syslog.org