in reply to Perl Memory problem ...
If you give us more information we should be able to help. Meanwhile, these old nodes might give you a clue as to what your problem might be.
Using this node for Memory Tools References nowadays:
- Re^2: Perl, C and deallocation responsibility (Returning Memory back to the OS References) - Perl on Linux and Windows (unlike most other Perl platforms) does return "big" chunks of memory back to the OS
- Losing my memory
- Perl Memory problem ...
- memory leak
- Memory leak detection
- Timing of garbage collection
- Memory is not released back to operating system
- How to return unused memory to OS? (my reply)
- demonstrate that perl can give back memory to the OS
- Not able to release memory
- Out of memory during "large" request
- Memory Leak when slurping files in a loop
- Another "out of memory!" problem
- Long running tasks, perl and garbage collection
- Memory leaks and circular references
- Valgrind, Perl, and banishing leaks
- Garbage Collection and undef
- undefining hashes to free memory
General Areas to consider when faced with a Memory Leak
- Stability: Once your process grows to a certain size, does it stay there, or does it grow further over time?
- Large CPAN Modules. Are you using any?
- Circular references. See Eliminate circular reference memory leak using weaken (Perl Maven).
- Scoping.
- File/Database handling.
- Tools. Try one or more from the following section.
Perl Tools and Advice
- Profiling Memory Usage talk by Tim Bunce (YAPC::NA Austin Texas 2013 featuring Devel::SizeMe). Resident size vs vm size, rss, segments, /proc/pid/maps (e.g. SO maps question), malloc and the heap, perl internals, pads, constant folding, arenas, recursion, perl data structures (lots of pointers), ... (update: this reply describes memory used by lexical variables in subroutines at the 09:20 minute mark)
- Plans for adding memory profiling to Devel::NYTProf
- Devel::NYTProf
- Devel::Gladiator
- Devel::LeakTrace
- Test::LeakTrace
- Devel::Leak
- Devel::FindRef
- Devel::Cycle
- Test::Memory::Cycle
- Devel::MemoryTrace::Light
- Devel::SizeMe by Tim Bunce (mentioned in talk above)
- Devel::Size
- Devel::DumpSizes
- Devel::Arena
- Devel::Mallinfo
- Devel::Peek
- splain (diagnostics, splain - produce verbose warning diagnostics)
- perldiag (search for "out of memory", "malloc", etc)
- perlvar (search for "$^M")
- perlfaq3 (search for "malloc", "memory", ..., perl -V:usemymalloc displays value of "usemymalloc" configuration variable)
- How can I free an array or hash so my program shrinks? (perlfaq3)
- How can I make my Perl program take less memory? (perlfaq3)
- INSTALL (Build and Installation guide for perl 5, ... search for malloc)
- perl malloc.c source code
- Perl::Critic - static code analysis for Perl
- MCE::Shared
- Mini-Tutorial: Perl's Memory Management by ikegami
- As pointed out by dave_the_m: "once the perl interpreter has exhausted all memory, there's very little it can do. There is a reserved fixed buffer to allow it to display the OOM error itself, but doing anything complex like displaying a Carp::confess()-like stack backtrace would require many memory allocations to generate the display, all of which would themselves fail".
General Tools
- Memory leak (wikipedia)
- Memory debugger (wikipedia - contains a list of memory debugging tools)
- Top 20 list of memory leak detection tools (softwaretestinghelp)
- Valgrind (see also: Test::Valgrind mentioned here)
- AddressSanitizer (wikipedia)
- VTune memory profiler (wikipedia) - a performance analysis tool for x86 based machines running Linux or Microsoft Windows operating systems
- Clang (Clang is a compiler front end for the C, C++, Objective-C, and Objective-C++ programming languages ... includes a static analyzer, and several code analysis tools ... Clang operates in tandem with the LLVM compiler back end)
- LLVM (LLVM is a set of compiler and toolchain technologies that can be used to develop a front end for any programming language and a back end for any instruction set architecture ... designed around a language-independent intermediate representation (IR) that serves as a portable, high-level assembly language that can be optimized with a variety of transformations over multiple passes)
- Boehm garbage collector aka Boehm–Demers–Weiser garbage collector (wikipedia)
Static Code Analysis
- Static code analysis (wikipedia)
- List of tools for static code analysis (wikipedia)
- CodeSonar static code analysis (wikipedia)
- Coverity static code analysis (wikipedia)
- Dynamic program analysis (wikipedia)
- Duplicate code (wikipedia)
- DRY (wikipedia)
- Rule of three (wikipedia)
Code Profiling and Optimization
Testing Concurrent Software
Security Testing
- Re: Security techniques every programmer should know (Security References) (long list of Security references)
Some Related PM Nodes
- Is $^M a leftover April Fool? by BrowserUk (2003)
- How use $^M? by xiaoyafeng (2021)
- Make Perl use real malloc by creamygoodness (2011)
- Re^2: Do subroutine variables get destroyed? (Deterministic Destructor References) (discusses deterministic destructors, tracing vs reference-counted garbage collectors, RAII, ...)
- Re^2: A short whishlist of Perl5 improvements leaping to Perl7 (compares Perl's simple lexical scoping with Python's global and nonlocal keywords)
More Nodes Added Later
- Small Hash a Gateway to Large Hash? by lsherwood (2014)
- write hash to disk after memory limit by hailholyghost (2015)
- Do subroutine variables get destroyed? by bt101 (2016)
- Perl and Garbage Collection by Svetlana (2016)
- Perl "Out of memory!" Error signal backtrace by OnEhIppY (2021)
- Memory efficient design by harangzsolt33 (2022)
- Memory usage double expected by sectokia (2022)
- Re: FFI::Platypus: Replace malloc with GC_MALLOC? by ikegami (2022)
- Re^3: FFI::Platypus: Replace malloc with GC_MALLOC? by me (2022)
- Perl, C and deallocation responsibility by SheWolf (2023)
- Re^2: Perl, C and deallocation responsibility (Returning Memory back to the OS References) by me (2023)
- Re: Tk-804.036 build failure by syphilis (2023) - 32-bit build ok, 64-bit build crashes (also gcc v clang)
- Re^3: Extracting selected fields form file record (don't use File::Slurp) by me (2023) - see haukex reply
- Does Perl have a baked-in memory llimit? by cormanaz (2023) - out of memory File::Slurp-ing a half-gig file, with 32 GB of system memory (32 v 64 bit perl build?)
- Memory limitation of a perl process in Unix by asinghvi (2004) - out of memory: is there a memory limit on a perl process on Unix/Solaris?
- Re^5: Is there something faster than string concatenation? (hermit crabs) by me (2007) - performance of Perl string concatenation depends on performance of the underlying C library realloc function
- Strawberry: Both IO and global match are VERY SLOW. Unless pre-heated (but why?) by anonymonk (2023) - see replies by syphilis, Discipulus and tonyc
See Also
Updated: Many extra references were added long after the original reply was made.
|
---|
In Section
Seekers of Perl Wisdom