Re: Looking for Leaks in all the wrong places
by Matts (Deacon) on Jun 09, 2002 at 20:21 UTC
|
This is a tough one. Finding memory leaks in Perl code is *really* hard. Unfortunately Devel::Leak (and Apache::Leak which is based on the former) are next to no use whatsoever, because they tend to tell you only that you have a leak, not what caused it.
You may find some wisdom in the mod_perl guide section on Debugging. But I wouldn't hold out too much hope there.
My personal solution to detecting memory leaks in mod_perl scripts (and which I used with good success in finding memory leaks in AxKit, is to use what I guess is a simple step through system, with a slight difference.
First of all it's important to note how AxKit is structured (and try and apply it to your code if you can).
In AxKit, the entire request is wrapped in an eval{} block. At the end of the eval block if I detect an error I can handle it appropriately, depending on what kind of error the user has and what configuration is set. But don't worry too much about that - the important thing is the eval{} block. Even if you don't write the error handling bit, you can debug this just by using that eval{} block and ignoring that your script might return something invalid for now.
Now in order to debug my memory leaks, I first find a URI that causes the leak on every request. Then I set apache running in single thread mode (/path/to/apache/bin/httpd -X). Then I hit the page with apachebench (/path/to/apache/bin/ab). I check the process is leaking by watching it in another window running "top". Following so far? Good.
Now, I add a simple: die "Leak test"; to the start of my eval{} block (just inside the opening curly bracket). Stop/start apache (still in single thread mode). And hit it again with apachebench. Your request should not leak here. If it does, you know the problem is in your eval handling code or before you got to your die().
Now move your "die" further into your code, following the path you know this request takes. Use fairly coarse steps at first, and when you get into the leak, step back in single lines.
I guarantee you that eventually you will discover where your leak is.
Now of course fixing it, that's another matter... The things that are the worst culprits for memory leaks in perl code are:
I hope that's some help. I feel deep sympathy for you as finding memory leaks is no fun at all. Good luck! | [reply] [Watch: Dir/Any] [d/l] |
|
| [reply] [Watch: Dir/Any] |
|
Thanks Matts! And hey, I really enjoy using all your XML stuff too.
The process for identifying memory leaks looks like a real pain in the butt. After thinking about it for a while, I would be perfectly happy to just automatically reboot the server after CPU / Memory are depleted to a certain point.
So my new question is: What combination of CPU utilization / memory paucity should I use as the trigger to automatically reboot?
My system is a single processor Linux box with 750M of memory running Apache and Oracle with mod_perl.
~~~~~~~~~~~~~~~
I like chicken.
| [reply] [Watch: Dir/Any] |
|
The process for identifying memory leaks looks like a real pain in the butt. After thinking about it for a while, I would be perfectly happy to just automatically reboot the server after CPU / Memory are depleted to a certain point.
this is a really bad attitude to take towards your code. not only has matts given you great code which you compliment, but he's given you an outline for advanced debugging techniques he used in the same code you praise.
instead of taking the opportunity to learn more about bugs that might be in your code (and software development in general) from someone who took the time to provide an detailed answer your question, you've decided to throw away his proven advice.
if you feel his advice is over your head, and you would like to learn but don't think you're capable yet, say so. i'm sure other suggestions can be offered, including advice on your follow-up question. as it stands, your dismissive tone leaves me unsympathetic to your current situation.
why would anyone here want to provide you with help in the future if you openly throw away such good advice?
i certainly wouldn't.
~Particle *accelerates*
| [reply] [Watch: Dir/Any] [d/l] |
A reply falls below the community's threshold of quality. You may see it by logging in.
|
|
You should *never* need to reboot the whole server - instead set MaxRequestsPerChild low (axkit.org has it at 100), and install something like Apache::SizeLimit to make sure you don't get one particular process going totally out of control.
| [reply] [Watch: Dir/Any] |
|
I agree with particle on this one. Who is to say that tomorrow you won't write another script with exactly the same problem your current one has? In that case, your server resources will become depleted twice as fast without an end in sight. Also, why be satisfied with code that you know is broken?
thor
| [reply] [Watch: Dir/Any] |
Re: Looking for Leaks in all the wrong places
by gav^ (Curate) on Jun 09, 2002 at 23:56 UTC
|
If you are running Apache, you may find it nice to have a cron job that restarts it every 24 hours or so. I've always did this because it sounded like a sensible thing to do, but it wasn't until recently I found out why :) We had a server which apache was never restarted because it sometimes forked a long running child process that would also be killed. As a "quick fix" the cronjob was turned off. Weeks later, the machine was out of memory and wierd things started happening due to it having 8 Apache child process each using 25-27 meg of memory.
Often debugging a memory leak isn't a viable option, the main use of this box was to run an app which is a big ball of mud and contains > 7,000 lines of frightening perl.
gav^ | [reply] [Watch: Dir/Any] |
|
Better yet, use Apache::SizeLimit which can handle this problem automatically. Adam Robinson just added support for Apache::SizeLimit to Bricolage and the results have been impressive.
-sam
| [reply] [Watch: Dir/Any] |
|
Yes, this is a perfect job for Apache::SizeLimit. And I'm not just saying that because I'm the maintainer. No one should run mod_perl without it (or its cousin Apache::GTopLimit).
| [reply] [Watch: Dir/Any] |