Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

Re: Perl Memory problem ...

by davido (Cardinal)
on May 10, 2020 at 16:04 UTC ( #11116652=note: print w/replies, xml ) Need Help??


in reply to Perl Memory problem ...

Perl won't release memory back to the operating system at runtime, only on global destruction as the script terminates. However, Perl will re-use the memory that it has allocated and that it is no longer using. For example, if you declare a lexical variable in one scope and load it up with 30MB of data, but then let that variable fall out of scope, Perl should be able to re-use that memory space.

Places to look:

  • Circular references: These can cause memory leaks since the memory never has its reference count drop to zero, Perl can't re-claim it for internal use.
  • Modules: Are you using half of CPAN? There's nothing wrong with using Moose, DBIx::Class, and many of the other slightly heavy distributions on CPAN, but they do use memory.
  • Scoping: Are you allowing lexical variables to fall out of scope? (You should be.) Program with the objective of keeping state as ephemeral as possible; let variables have the narrowest possible scope - particularly those that are holding large chunks.
  • File/Database handling: Are you slurping in large files or large Database query data sets? Process smaller chunks at a time.
  • Stability: Once your process grows to 30MB, does it stay there, or does it grow further over time? Frankly, even if it's consuming 100MB, that's not the end of the world as long as you can count on it staying close to that plateau. If it keeps on growing over time, you've got a bigger problem (a memory leak). A well-behaved process that hits a memory plateau and stays there is usually acceptable.
  • Tools: Try valgrind, Devel::LeakTrace, Devel::MemoryTrace::Lite, and other such tools can help you to watch how memory is growing and being used.

Dave

Replies are listed 'Best First'.
Re^2: Perl Memory problem ...
by eyepopslikeamosquito (Bishop) on May 10, 2020 at 20:38 UTC

    Perl won't release memory back to the operating system at runtime, only on global destruction as the script terminates.
    While that may be true for most practical purposes, I believe Perl simply defers to the implementation of the user-level malloc function - see for example: Memory is not released back to operating system.

    Last time I looked, both Linux and Windows malloc do return "large" chunks back to the OS.

      Thanks. I was probably misremembering Tim Bunce's YAPC talk: https://www.youtube.com/watch?v=GIIeOntmojg. I'll have to review it again when I have a moment. Your research looks thorough, though.

      Update: At about 4:10 Tim discusses malloc behavior, and suggests that larger allocations are mmapped, and the gist seems to be that while rare for malloc to actually give back, it is more likely for those larger mmapped chunks. So, thanks for motivating me to find that talk again. I'll have to watch the rest of it again this evening. It does confirm your assertion, too.


      Dave

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11116652]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (2)
As of 2021-09-28 20:11 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?