Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Auto-restarting script if it dies

by mojodaddy (Pilgrim)
on Mar 09, 2007 at 12:24 UTC ( [id://603978]=perlquestion: print w/replies, xml ) Need Help??

mojodaddy has asked for the wisdom of the Perl Monks concerning the following question:

Greetings Monks,

Since I am ridiculously new to programming of any kind, scarcely an hour goes by where I am not reminded of the vastness of my ignorance. Still, I've inexplicably fallen in love with perl, so I have no choice but to continue trying to do things I have no idea how to do.

Most recently I set out to find a way to ensure that if a script dies, it is restarted automatically. That way I can go to sleep or maybe even leave the house once in a while. So I dove into perldoc, CPAN, Perl Monks, Google, the Ubuntu forums (I am only marginally less new to Linux than I am to programming), etc., and found loads of stuff relating to PIDs and daemons (I know it's almost baseball season because I keep finding myself typing "damon" instead of "daemon") and forks and parents and children and executing perl scripts from within bash scripts, all of which I found to be either insanely over my head or infuriatingly vague (no offense). I suppose it's like Googling for instructions on how to open the lid of your laptop. It's so obvious that no one thinks to write a tutorial on it.

But when I saw File::Temp on CPAN, a little light went on. All I'd been looking for was simple a way for the script to say "Yes, I'm running," or "No, I'm not running." Why monkey around with a PID or a bash script if I don't need (or know how) to?

So using File::Temp my script (the Script) creates a temporary file when it starts running and which is removed when it stops running:

#!/usr/bin/perl # always_be_running.pl use strict; use warnings; use File::Temp (); # creates a temporary file in /tmp which is removed + when the script exits my $fh = new File::Temp(); my $fname = $fh->filename; # now open another file and write the name of the temp file to it my $checkme = "/home/mojodaddy/checkme.txt"; open CHECKME, "+>$checkme" || die "Can't open checkme file: $!\n"; print CHECKME $fname; close CHECKME; print "Script says: 'Hey, I just started running!'\n"; sleep 60; # or whatever print "Script says: 'Boy, I'm bushed! I'm gonna go lie down.'\n";
Then I create and execute a second script (the Checker) which starts the Script, then looks to see if the temp file is there. If it is, it goes to sleep. If not, it restarts the Script:
#!/usr/bin/perl # check_if_its_running.pl use strict; use warnings; my $line; my $checkme = "/home/mojodaddy/checkme.txt"; start_it(); while (1) { open CHECKME, "<$checkme" || die "Couldn't open checkme file: $!\n +"; while (<CHECKME>){ # get the line with the name of the tempfile $line = $_; } close CHECKME; if (-e $line){ # see if the file exists print "(Checker says: 'Woo-hoo! Script is still running!')\n"; sleep 10; } else{ print "(Checker says: 'Doh! Script stopped running! I'd better + restart it.')\n"; start_it(); } } sub start_it { system("perl always_be_running.pl"); }
And it works!

The only peculiar thing about it is that the line "Woo-hoo! Script is still running!" never prints, but when I hit CTRL-C to exit the script, then it prints, and keeps printing every 10 seconds until I hit CTRL-C again. Now that's a bit of a head-scratcher... but did I mention it works?

I welcome your comments and suggestions.

UPDATE:
Once again I want to thank everyone who took the trouble to respond. I also wanted to mention for the record that when I implemented this on my real script rather than the test script above, and then waited for it to crash (which did not happen for several hours), I obsesrved the same odd behavior as when the "Woo-hoo!" line wouldn't print; only this time there was nothing to print, so instead I just saw the last line of output followed by a blinking cursor; I did a ctrl-c which brought me back to my bash prompt. So it's seeming less and less like a print buffer issue.

The fix? I don't know why I didn't just set it up this way in the first place, but rather than have the Checker script invoke the Script initially, I simply start the Script myself in the normal way; I remove the first call to start_it() in the Checker, and then start it as a background process. I was delighted to find, another few hours later, that the first instance had died, but that another instance had started up again in a separate terminal window and was whirring merrily away.

Happy, happy days. :)

Replies are listed 'Best First'.
Re: Auto-restarting script if it dies
by samizdat (Vicar) on Mar 09, 2007 at 13:04 UTC
    Hi, mojodaddy!

    I'm not awake enough to deal with buffering of print statements, but I will compliment you on re-inventing the concept of a pidfile. Good going! {and, I don't mean that facetiously! :-}

    There's another cool concept in UN!X and BSD and Linux called "respawning" in init. In Linux (since you mentioned that) you just find the little file called /etc/inittab and add a line at the end of it with your program name, like so:

    my:2345:respawn:/usr/local/bin/myprog.pl -m -y --options

    where 'my' is a token up to three characters, '2345' says run it in all Runlevels from multi-user to X GUI, 'respawn' says 'restart if it dies', and the rest is the path to your executable program and its options.

    To kick it off, either reset the init process with kill -1 1 or restart your computer. This is a simplified description, but the result is that the init process will do all your restarting work for you. You won't have direct writing of stderr or stdout to a terminal there, but you can learn how to associate your process with a running program or with syslogd and there you go, off on another exploration! :D

    Don Wilde
    "There's more than one level to any answer."
      Cool, thanks. And as for your simplified description, that's the only kind I'm likely to understand! : )
Re: Auto-restarting script if it dies
by merlyn (Sage) on Mar 09, 2007 at 14:19 UTC
    My DBI Logger helper script needs to be running all the time, so I wrote a watchdog that sits above it. If the heavy lifting child ever exits, the parent restarts it immediately. And since the parent doesn't do much besides just sit there, it probably won't go away.
      On Unix, daemontools provides a fairly straightforward way to do exactly what merlyn describes without writing any code. It goes a step further, and has one additional process monitoring all of the monitoring process, with that process restarted by init. That's pretty foolproof, and it's straightforward to set up many monitored services.
        As a matter of fact, I spent a few hours grappling with daemontools, and it's a perfect example of the freakishly opaque "explanations" I described in my original post. Here's its FAQ entry for the question "How do I create a service directory?":

        Answer: The only required component of your service directory is an executable file, ./run, that runs your daemon in the foreground, exiting when your daemon exits.

        Huh? Does that mean I'm supposed to create ./run? And if so, what is it? And how, exactly, do I create it? The FAQ says:

        Typically ./run is a shell script. For example:

        #!/bin/sh
        echo starting
        exec clockspeed

        "Typically?" So is my situation typical? How can I tell? And if ./run is a shell script in a typical case, what is it in other cases?

        So let's assume that I am supposed to write a shell script (which in my case would execute my perl script?) and save it as ./run in my service directory. But if so, why not just say so? Plus, would it kill them to explain how? It's like if I said, "How is a moon-rock different from a regular rock?" and they said "OK, first go to the moon and get some rocks, and I'll tell you the rest when you get back..."

        Of course the reality is that if the daemontools FAQ had been anything less than completely baffling, I wouldn't have written my solution, in which case I would never had received a personal response from merlyn! So it's all good. : )

      Wow... You'll have to forgive me for being a little star-struck here. I'm both hideously embarrassed and oddly proud to know that you've read and responded to my humble code. (But mostly the former.) Thanks, merlyn.
        This could be just me, but it seems a bit odd to start one script, and then start another script to check the first script.

        Perhaps you could in the checker.pl delete the checkme.txt file (under the line where you declare var $checkme, that way, on start up it's deleted and does not pick up an old version of the file with an old temp file name, as checkme.txt is never deleted any where in your code)

        Run the checker.pl script, first time in the loop, file does not exist, starts always_running.pl and creates checkme.txt and then the checker script continues checking checkme.txt.

        Just trying to refine the process, I liked the idea lots.

        Ant
Re: Auto-restarting script if it dies
by zentara (Archbishop) on Mar 09, 2007 at 13:41 UTC
    Here is a little fun script, as an alternative to a temp file. I got the idea from Can't catch signals after an exec?
    #!/usr/bin/perl use warnings; use strict; use POSIX(); $|=1; END { exec $0 } #needed to catch internal die's my $sub = sub { exec $0 }; # POSIX unmasks the sigprocmask properly my $sigset = POSIX::SigSet->new(); my $action = POSIX::SigAction->new( $sub, $sigset, &POSIX::SA_NODEFER ); POSIX::sigaction(&POSIX::SIGINT, $action); POSIX::sigaction(&POSIX::SIGUSR1, $action); POSIX::sigaction(&POSIX::SIGTERM, $action); # can still be killed with a kill -9 or SIGKILL # but kill -15 or SIGTERM will restart my $counter = 0; while(1){ print ++$counter,"\n"; if($counter == 4){die} sleep 1; } __END__

    I'm not really a human, but I play one on earth. Cogito ergo sum a bum
Re: Auto-restarting script if it dies
by EdwardG (Vicar) on Mar 09, 2007 at 13:25 UTC

    -e doesn't need you to open and read the contents of the file. It's purpose is to test for the existence of a file, like this:

    if (-e $checkme) { print "woo hoo\n"; }

    And what about the script that checks check_if_its_running.pl is still running?

    Here's a meta solution ;-)

    for (my $i = 2 to MAX_INTEGER) { if (-e "$checkme.$i") { print "woo hoo\n"; } else { start_it($i); } }

     

      Thanks for that. Maybe I misunderstood your response, but $checkme isn't the tempfile, it's the file with the name of the tempfile written in it. That file is going to exist no matter what. (Isn't it?) And since the checker script doesn't know the name or filehandle of the tempfile, it can't go looking for it, so that's why I create a file whose name it does know, wherein it will find the name of the tempfile, whose existence it can then verify. So -e is checking the existence of $line, which in this case is the name of the tempfile as read from $checkme.

      However, MAX_INTEGER sounds intriguing. I'm going to have to look that up... : )

Re: Auto-restarting script if it dies
by former33t (Scribe) on Mar 09, 2007 at 14:05 UTC
    Try setting your STDOUT buffer to autoflush. I didn't try your exact implementation, but that looks like the problem to me.

    $| = 1;

      I wasn't sure whether you meant I should add that line to the Script, the Checker, or both -- so I tried all three! The line still didn't print, though. Which is fine, because in my actual implementation I don't need to print a line there. But it is curious, isn't it? Thanks for your reply, though.
Re: Auto-restarting script if it dies
by naxxtor (Initiate) on Mar 13, 2007 at 04:00 UTC
    If you just need the script to restart if it exits (for whatever reason) then could you not write a very (very) simple shell script to wrap it? Something like:
    while [ 1 ]; do ./myscript.pl done
    and pop that into run.sh (or whatever), and use that to run the program. You could even run it as a background task from there... set and forget.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://603978]
Approved by Corion
Front-paged by moklevat
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others goofing around in the Monastery: (4)
As of 2024-03-19 11:03 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found