Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Mod_perl, Memory Leaks and WeakRef

by djantzen (Priest)
on Dec 25, 2003 at 00:22 UTC ( #316940=perlquestion: print w/replies, xml ) Need Help??

djantzen has asked for the wisdom of the Perl Monks concerning the following question:

Hi All, Well it looks like jolly St. Nick brought me a memory leak this year. It was something I had a good idea was coming and thought I had a handle on, but apparently not.

What I've got is a set of modules serving as the backend for a CGI::Application-based script running under mod_perl. The trouble is that the modules create datastructures with lots of circular references. Essentially I've got a directed graph implemented as an adjacency list so edges and vertices refer to one another. Furthermore, each vertex in the graph contains a tree where parent and child nodes each hold a reference to one another as a doubly-linked list.

My plan was to use WeakRef to break these circular references, and it seems to work in cases such as my global object cache and in the tree nodes. However, in other cases where the "forward" and "back" references are not so clear, i.e., in my graph edges and vertices, I've run into premature garbage collection leading to some extremely vexing "can't call methodX on an undefined value" errors. So I've removed the trouble making weakenings, but now it looks a lot like I've got a memory leak under mod_perl. My tools for figuring this are admittedly crude, looking at the output from 'top' and watching the memory addresses increase on each invocation of the script. I'm trying to use Apache::Leak as well (calling it in my CGI like leak_test { $my_app->run() };) but it dumps core before providing any useful information: "FILE = 'Wed Dec 24 19:07:45 2003 notice child pid 24391 exit signal Segmentation Fault (11), possible coredump'"

Now I'm very new to Web programming, so perhaps I'm missing an obvious answer, but I need some help solving this. So far I've tried manually calling DESTROY on all my objects in my implementation of CGI::Application::teardown, which, to my surprise, appears not to work. I thought for sure the only drawback was the ugliness of brute-forcing the solution, but apparently I'm missing something about perl's GC. I'm wondering now if I need to traverse my datastructures and manually break those circular references. This would have added benefit since further usages of these modules will face the same problem in some imaginable contexts, so writing code to perform it reliably could come in handy down the road. Or is there a way to tell mod_perl to blow away all object references in a process when the CGI request ends without requiring recompilation of the files on every invocation?

Thanks much, David


"The dead do not recognize context" -- Kai, Lexx

Replies are listed 'Best First'.
Re: Mod_perl, Memory Leaks and WeakRef
by cees (Curate) on Dec 25, 2003 at 03:21 UTC

    It's funny that you should ask this question today, because just this morning I learned about a module that you can use to automatically find and weaken circular references. Have a look at the Data::Structure::Util module, specifically the has_circular_ref and circular_off methods.

    Most of you will know that I learned this from the very informative Perl Advent Calendar.

    - Cees

      Cool! This looks like just what I need. Too bad I'm such a scrooge or I might have read the Advent Calendar :^P


      "The dead do not recognize context" -- Kai, Lexx
Re: Mod_perl, Memory Leaks and WeakRef
by Joost (Canon) on Dec 25, 2003 at 00:55 UTC
    I'd try to avoid weak references if at all possible.

    Some remarks:

    • Calling DESTROY on an object to force it to destruct doesn't work at all as far as I know, but someone else might know more than I do.
    • I would suggest you take a look at Graph::Directed. It keeps the graph as a seperated object from the nodes and edges, which should remove the need for references in the nodes and edges themselves.
    • You can also try writing your own graph object that removes the references in the nodes/edges when it goes out of scope. That way you also don't need weak references.
    HTH,
    Joost
      Well you're right about DESTROY; after rereading perlobj I see it doesn't actually cause any reference destruction, but just gives the programmer a last chance to do some custom cleanup. So that's out. As for Graph::Directed, it utilizes an adjacency matrix which is why it doesn't introduce the reference problems I'm encountering. I chose to write my own graph class utilizing an adjacency list though since the graph itself must be persisted in a database, and building a table with an 'origin' => 'destination' mapping seemed more natural than trying to store a matrix or translate between the two implementations when moving between disk and memory.

      Your last suggestion may well work, although I'll have to make some changes to the way I treat my graph object references. Actually perlobj suggests basically the same thing: "the best solution is to create a non-recursive container class that holds a pointer to the self-referential data structure. Define a DESTROY method for the containing object's class that manually breaks the circularities in the self-referential structure."

      thanks for the input davido.


      "The dead do not recognize context" -- Kai, Lexx
Re: Mod_perl, Memory Leaks and WeakRef
by PodMaster (Abbot) on Dec 25, 2003 at 07:43 UTC
    You don't call DESTROY, you just undef your object, and break all the circular references in DESTROY.

    MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
    I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
    ** The third rule of perl club is a statement of fact: pod is sexy.

      Of course I still have to find all of the references to the object otherwise I'll just decrement the refcount of the object and it won't be destroyed. Point taken though about the role of DESTROY versus undef.


      "The dead do not recognize context" -- Kai, Lexx
Re: Mod_perl, Memory Leaks and WeakRef
by diotalevi (Canon) on Dec 26, 2003 at 01:06 UTC
    How about writing a module for yourself that given and inventory of objects will examine perl's arenas and tell you which ones still exist? You might be able to see what parts are leaking then.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://316940]
Approved by davido
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (3)
As of 2020-06-07 03:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Do you really want to know if there is extraterrestrial life?



    Results (42 votes). Check out past polls.

    Notices?