This is a tough one. Finding memory leaks in Perl code is *really* hard. Unfortunately Devel::Leak (and Apache::Leak which is based on the former) are next to no use whatsoever, because they tend to tell you only that you have a leak, not what caused it.

You may find some wisdom in the mod_perl guide section on Debugging. But I wouldn't hold out too much hope there.

My personal solution to detecting memory leaks in mod_perl scripts (and which I used with good success in finding memory leaks in AxKit, is to use what I guess is a simple step through system, with a slight difference.

First of all it's important to note how AxKit is structured (and try and apply it to your code if you can).

In AxKit, the entire request is wrapped in an eval{} block. At the end of the eval block if I detect an error I can handle it appropriately, depending on what kind of error the user has and what configuration is set. But don't worry too much about that - the important thing is the eval{} block. Even if you don't write the error handling bit, you can debug this just by using that eval{} block and ignoring that your script might return something invalid for now.

Now in order to debug my memory leaks, I first find a URI that causes the leak on every request. Then I set apache running in single thread mode (/path/to/apache/bin/httpd -X). Then I hit the page with apachebench (/path/to/apache/bin/ab). I check the process is leaking by watching it in another window running "top". Following so far? Good.

Now, I add a simple: die "Leak test"; to the start of my eval{} block (just inside the opening curly bracket). Stop/start apache (still in single thread mode). And hit it again with apachebench. Your request should not leak here. If it does, you know the problem is in your eval handling code or before you got to your die().

Now move your "die" further into your code, following the path you know this request takes. Use fairly coarse steps at first, and when you get into the leak, step back in single lines.

I guarantee you that eventually you will discover where your leak is.

Now of course fixing it, that's another matter... The things that are the worst culprits for memory leaks in perl code are:

• Closures - this is a particular concern for mod_perl since the closure will sometimes remain active for the lifetime of the process. Fixing closure related memory leaks requires an understanding of what a closure is and what it's for and how and why it holds onto an instance of your variable.
• Circular references - these nasty little buggers mean your variables never get freed. There are techniques for avoiding them, and almost always they are unnecessary.
• Perl bugs - yes there are such things. Like for example in 5.00503, the following code (which really is a think-o on my part for writing it in the first place as it makes no real sense) leaked like a seive:

  my @foo = get_list() if $condition;
  [download]

  (or it was something like it - I don't fully remember the exact details now, they are lost in the anals of axkit history)
• XS modules - sometimes an author of an XS module will forget to free some memory, or mess up the refcount of a variable. It's very easy to do - I should know ;-)

I hope that's some help. I feel deep sympathy for you as finding memory leaks is no fun at all. Good luck!