Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Perl Daemons

by pbeckingham (Parson)
on Jul 28, 2004 at 17:30 UTC ( [id://378125]=perlquestion: print w/replies, xml ) Need Help??

pbeckingham has asked for the wisdom of the Perl Monks concerning the following question:

I am often tempted to create daemons in Perl - programs that essentially run forever*. Examples include various listeners and monitors.

Most of those programs can, and generally are, rewritten by me to run from the crontab, so I can let the OS do the waiting, and my program doesn't have to loop, block or wait. The real benefit of this is that my program exits and returns resources to the OS. It never gets the chance to leak, bloat, and become a burden. Memory allocated to Perl objects may be released to be reused by other Perl objects, but never gets released back to the OS.

But sometimes there is a need for a program to stay resident in memory. Such a program must not leak memory or bloat. In fact it must flatline its resource usage in order for it to continue its existence - long term.

While this holds true for any language in which I implement a daemon, it seems that Perl is particularly unsuited. Have any of you dealt with this issue successfully? Because I sure haven't.



* Forever in this case means until kernel upgrade.

Replies are listed 'Best First'.
Re: Perl Daemons
by Joost (Canon) on Jul 28, 2004 at 17:43 UTC
    Perl does return memory to the OS, just not all of it all the time. :-)

    Anyway, I'd go for the solution the apache webserver uses: run a really simple controlling process, that fork()s off children to do the real work, and kill them after handling a certain number of requests. That way, you only have to worry about bloat in the controlling program. In fact, you can even use code in the children that you KNOW are not safe in the long run.

    update: typo fixed ("the solution")

      I agree with you - essentially the program doing the real work has to exit periodically, or every time. The controller can often be reduced to a shell script, where the shell itself we assume is not suffering from the same problem.

        Perlmonks' streaming ticker periodically execs itself which is a variant on the same theme. Instead of forking off children to do work, the main POE process does all of the work and avoids such problems by restarting itself.

        I don't think the controller needs to be a shell script, in fact, I think that in a perl script it might be easier to control the memory usage (just a hunch). I think that as long as you don't create and destroy loads of lexicals in the controller, you should be alright.

        Ofcourse, if you have a simple controller, it should be easy enough to test for leaks - just do a couple of million iterations though the controller's mainloop without running the childs.

Re: Perl Daemons
by Your Mother (Archbishop) on Jul 28, 2004 at 19:26 UTC

    I wrote a web chat in Perl using techniques from Network Programming with Perl ISBN 0201615711. The chat daemon/server gets an extremely light load but it's up and running for six months at a stretch without trouble. Never had to restart it except when my host reboots the server. The book is really good. John Orwant jacket quote: ...a superb book...a must-have for the serious Perl programmer.

    Stein explains why some of the common idioms for things like CHLD/zombie reaping are flawed and can add up to problems over long periods. That might be why Perl has obtained a dicey rep(?). This also seems the right time to mention that I've had trouble with Proc::Daemon (not memory, but that it won't even start a daemon on OS X). Stein's techniques worked transparently for me on a couple *nixes. I started sketching out a CPAN module (or Proc::Daemon update) with some of Stein's code but I never got it working the way I wanted as a standalone or writing to syslogs well enough to publicize.

Re: Perl Daemons
by paulbort (Hermit) on Jul 28, 2004 at 18:04 UTC
    Don't worry about making programs run forever; worry about people who think that servers run forever. The same memory leaks and stealth bugs that might be in your Perl might also be in the server OS.

    I don't trust a server with more than about 30 days of uptime. In fact, the Big Brother server monitoring software includes code to send an alarm if server uptime exceeds user-defined threshholds.

    Worry about making the code solid for a 30-90 day run. Any longer than that and you could run into problems with the parts you didn't write.

    The worker child suggestion above is also a very good idea.

    --
    Spring: Forces, Coiled Again!
      Maybe one should use another server monitoring software ;.)

      At my company we are using mon, a fantastic peace of software to monitor more than fifty devices.

      Mon is a perl script (sic!) and the last time I bootet the server was January 1 this year, because we had to take away the power of the whole server room. Before that the server was running for more than 400 days and mon.pl is only being restartet when there are any config changes - I think I should even be able to reload the config with a SIG HUP, but didn't test it already.

      The server is running SuSE Linux 8.0 with a 2.4 kernel.

      Therefore I do not see any reason to restart a good written piece of perl daemon software just to avoid any potential trouble!

      Cheers Roland

        I'm sure that the OP can use care and write a solid, non-leaking daemon that would run for years. I did not mean to disparage OP's daemon, coding, or any monitoring software.

        I did mean that there are millions of lines of very clever code holding up that daemon, and it is impractical to test that code for months of constant operation. So as a precaution, monthly server restarts will do two things:
        1. Prevent any long-term leakage problems from rearing their ugly heads
        2. Catch any configuration changes that might have been made in the preceding month that would prevent the server from returning to normal operation. A failed restart at the beginning of a maintenance window is much easier to deal with than a failed restart after a move or extended power outage.

        --
        Spring: Forces, Coiled Again!
Re: Perl Daemons
by jsadusk (Acolyte) on Jul 29, 2004 at 02:37 UTC
    One way of making perl an effective daemon type language is to implement your daemon in the traditional UNIX manner (something that's kind of fallen out of style as of late). Have your listener be the same perl program that does the work, but when it gets a request, you fork(). Don't exec after that, just fork and handle the request. And then each request handler exits.

    You get paralel processing of requests. You get any code that would be likely to leak (possibly from variables not being garbage collected) exiting immediately after being done. And since you just fork and don't exec, you're not restarting the perl interpreter. If you're on a modern UNIX (linux, solaris, bsd, whatever), you have copy on write so you're not making a huge copy in memory of your entire perl process (this is especially the case if your listener process sits in a tight loop not updating much data).

    One drawback to this approach is getting information back from the handler processes. You can use pipes, or named pipes, but then you're limited to a stream of text that needs to be parsed. But if your handlers are mostly atomic, this is a non-issue.

    Another drawback is if you have a ton of requests, you have a ton of processes, and your system scheduler and VM start slowing down. Only a drawback compared to a threaded system however.

    Still, for medium load daemons with medium amounts of communication, this approach works extremely well and I've used it multiple times.

Re: Perl Daemons
by shenme (Priest) on Jul 29, 2004 at 01:36 UTC
    While I can't go into too much detail about what they do, I run several Perl programs as daemons because they must react near immediately. They process FTP upload traffic, email, pages, logs, interact with databases, watch for weirdness, etc. They've been running since Sept 2003. One has processed 45K transactions, another 365K, from just January thru June.

    Four out of the five below haven't grown a bit. The fifth has grown about 1.5MB since September, and that's called a bug. There are tools to help with bugs like that.

    (Highly boiled down 'ps' output to not mess up web pages)
    
      PID  PPID   VSZ   STARTED                  COMMAND
     2743     1   4200  Thu Sep 11 17:51:08 2003 budgdmon.pl
          budgdmon: waiting
     2803     1   5044  Thu Sep 11 17:51:09 2003 budgftpd.pl
          budgftpd: waiting
     2850     1   4308  Thu Sep 11 17:51:12 2003 pageqrdr.pl
          pageqrdr: waiting
     5810  5807   7040  Thu Sep 11 18:11:31 2003 perl
          perl -w /opt/budgie/admin/xflogger/xfsuck.pl -v -d 
                --database test --table XferlogX 
                --fromtail /var/log/xferlog
     6815  6814   4420  Thu Sep 11 18:17:03 2003 perl
          perl ModemMonitor.pl
    
    Now not all the processing takes place within the daemons. They do spawn off tasks for lengthy and/or nasty processing, so that probably contributes to the lack of problems _within_ the daemons.

    My take - Perl is "very fine" for writing daemons.

    --
    I'm a pessimist about probabilities; I'm an optimist about possibilities.
    Lewis Mumford

Re: Perl Daemons
by rdm (Hermit) on Jul 29, 2004 at 00:35 UTC

    A well written daemon can run 'forever' without causing problems. All you need to do is use a little care. A great example is an agent script that was written for the Cricket monitoring tool:
    CricketStats

    I've got decendents of this that have been running for months on some boxes!

    -Reality might not get out of Beta today. (O.Timas, "Bot")

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://378125]
Front-paged by diotalevi
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (5)
As of 2024-03-29 09:09 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found