pbeckingham has asked for the wisdom of the Perl Monks concerning the following question:
I am often tempted to create daemons in Perl - programs that essentially run forever*. Examples include various listeners and monitors.
Most of those programs can, and generally are, rewritten by me to run from the crontab, so I can let the OS do the waiting, and my program doesn't have to loop, block or wait. The real benefit of this is that my program exits and returns resources to the OS. It never gets the chance to leak, bloat, and become a burden. Memory allocated to Perl objects may be released to be reused by other Perl objects, but never gets released back to the OS.
But sometimes there is a need for a program to stay resident in memory. Such a program must not leak memory or bloat. In fact it must flatline its resource usage in order for it to continue its existence - long term.
While this holds true for any language in which I implement a daemon, it seems that Perl is particularly unsuited. Have any of you dealt with this issue successfully? Because I sure haven't.
* Forever in this case means until kernel upgrade.
Re: Perl Daemons
by Joost (Canon) on Jul 28, 2004 at 17:43 UTC
|
Perl does return memory to the OS, just not all of it all the time. :-)
Anyway, I'd go for the solution the apache webserver uses: run a really simple controlling process, that fork()s off children to do the real work, and kill them after handling a certain number of requests. That way, you only have to worry about bloat in the controlling program. In fact, you can even use code in the children that you KNOW are not safe in the long run.
update: typo fixed ("the solution")
| [reply] [Watch: Dir/Any] |
|
I agree with you - essentially the program doing the real work has to exit periodically, or every time. The controller can often be reduced to a shell script, where the shell itself we assume is not suffering from the same problem.
| [reply] [Watch: Dir/Any] |
|
| [reply] [Watch: Dir/Any] |
|
| [reply] [Watch: Dir/Any] |
Re: Perl Daemons
by Your Mother (Archbishop) on Jul 28, 2004 at 19:26 UTC
|
I wrote a web chat in Perl using techniques from Network Programming with Perl ISBN 0201615711. The chat daemon/server gets an extremely light load but it's up and running for six months at a stretch without trouble. Never had to restart it except when my host reboots the server. The book is really good. John Orwant jacket quote: ...a superb book...a must-have for the serious Perl programmer.
Stein explains why some of the common idioms for things like CHLD/zombie reaping are flawed and can add up to problems over long periods. That might be why Perl has obtained a dicey rep(?). This also seems the right time to mention that I've had trouble with Proc::Daemon (not memory, but that it won't even start a daemon on OS X). Stein's techniques worked transparently for me on a couple *nixes. I started sketching out a CPAN module (or Proc::Daemon update) with some of Stein's code but I never got it working the way I wanted as a standalone or writing to syslogs well enough to publicize.
| [reply] [Watch: Dir/Any] |
Re: Perl Daemons
by paulbort (Hermit) on Jul 28, 2004 at 18:04 UTC
|
| [reply] [Watch: Dir/Any] |
|
Maybe one should use another server monitoring software ;.)
At my company we are using mon, a fantastic peace of software to monitor more than fifty devices.
Mon is a perl script (sic!) and the last time I bootet the server was January 1 this year, because we had to take away the power of the whole server room. Before that the server was running for more than 400 days and mon.pl is only being restartet when there are any config changes - I think I should even be able to reload the config with a SIG HUP, but didn't test it already.
The server is running SuSE Linux 8.0 with a 2.4 kernel.
Therefore I do not see any reason to restart a good written piece of perl daemon software just to avoid any potential trouble!
Cheers
Roland
| [reply] [Watch: Dir/Any] |
|
| [reply] [Watch: Dir/Any] |
Re: Perl Daemons
by jsadusk (Acolyte) on Jul 29, 2004 at 02:37 UTC
|
One way of making perl an effective daemon type language is to implement your daemon in the traditional UNIX manner (something that's kind of fallen out of style as of late). Have your listener be the same perl program that does the work, but when it gets a request, you fork(). Don't exec after that, just fork and handle the request. And then each request handler exits.
You get paralel processing of requests. You get any code that would be likely to leak (possibly from variables not being garbage collected) exiting immediately after being done. And since you just fork and don't exec, you're not restarting the perl interpreter. If you're on a modern UNIX (linux, solaris, bsd, whatever), you have copy on write so you're not making a huge copy in memory of your entire perl process (this is especially the case if your listener process sits in a tight loop not updating much data).
One drawback to this approach is getting information back from the handler processes. You can use pipes, or named pipes, but then you're limited to a stream of text that needs to be parsed. But if your handlers are mostly atomic, this is a non-issue.
Another drawback is if you have a ton of requests, you have a ton of processes, and your system scheduler and VM start slowing down. Only a drawback compared to a threaded system however.
Still, for medium load daemons with medium amounts of communication, this approach works extremely well and I've used it multiple times.
| [reply] [Watch: Dir/Any] |
Re: Perl Daemons
by shenme (Priest) on Jul 29, 2004 at 01:36 UTC
|
While I can't go into too much detail about what they do, I run several Perl programs as daemons because they must react near immediately. They process FTP upload traffic, email, pages, logs, interact with databases, watch for weirdness, etc. They've been running since Sept 2003. One has processed 45K transactions, another 365K, from just January thru June.
Four out of the five below haven't grown a bit. The fifth has grown about 1.5MB since September, and that's called a bug. There are tools to help with bugs like that.
(Highly boiled down 'ps' output to not mess up web pages)
PID PPID VSZ STARTED COMMAND
2743 1 4200 Thu Sep 11 17:51:08 2003 budgdmon.pl
budgdmon: waiting
2803 1 5044 Thu Sep 11 17:51:09 2003 budgftpd.pl
budgftpd: waiting
2850 1 4308 Thu Sep 11 17:51:12 2003 pageqrdr.pl
pageqrdr: waiting
5810 5807 7040 Thu Sep 11 18:11:31 2003 perl
perl -w /opt/budgie/admin/xflogger/xfsuck.pl -v -d
--database test --table XferlogX
--fromtail /var/log/xferlog
6815 6814 4420 Thu Sep 11 18:17:03 2003 perl
perl ModemMonitor.pl
Now not all the processing takes place within the daemons. They do spawn off tasks for lengthy and/or nasty processing, so that probably contributes to the lack of problems _within_ the daemons.
My take - Perl is "very fine" for writing daemons.
--
I'm a pessimist about probabilities; I'm an optimist about possibilities.
Lewis Mumford
| [reply] [Watch: Dir/Any] |
Re: Perl Daemons
by rdm (Hermit) on Jul 29, 2004 at 00:35 UTC
|
A well written daemon can run 'forever' without causing problems. All you need to do is use a little care. A great example is an agent script that was written for the Cricket monitoring tool:
CricketStats
I've got decendents of this that have been running for months on some boxes!
-Reality might not get out of Beta today. (O.Timas, "Bot")
| [reply] [Watch: Dir/Any] |
|
|