memory usage and leakage

szabgab has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: memory usage and leakage by Sewi (Friar) on Jun 23, 2011 at 11:06 UTC
The VM value is the address space allocated by an application, not the memory used. It doesn't give any reasonable result at all. RSS is the memory actually used and this is not shared across instances. The sum of RSS of all processes is the memory (including swap) used on your computer. Add kernel memory, disk cache and free and you got all of your memory. If you want to share some memory, "use" all required modules and start threading, but you need to check if all modules are thread-safe and ensure your code to be thread-safe. If you care about CPU or memory usage, remove PEO and Moose from your project as they eat up half of the scripts memory. Try Padre - the free Perl IDE	[reply]
Re^2: memory usage and leakage by BrowserUk (Patriarch) on Jun 23, 2011 at 12:15 UTC
If you want to share some memory, "use" all required modules and start threading, Unless the application requires all the modules in all the threads, that is bad advice. It ensures that every thread will get it own copy of everything in every module, whether it ever uses it or not. And if the application does require all of those in every thread, it is badly designed and probably won't work anyway. If the OP structures his application properly, and requires only those modules he needs in each thread, it will reduce memory consumption conspicuously. But you're right that the simplest route to reducing memory consumption is to avoid behemoth frameworks that force you to load everything before you can do the simplest thing, and a shit-load more every time you want to do something extra. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]
Re^3: memory usage and leakage by Sewi (Friar) on Jun 23, 2011 at 15:14 UTC
You're right as long as the threads do different things. But he said, he'd need many instances, my suggestion was to make every "instance" one "thread". Try Padre - the free Perl IDE	[reply]
Re^4: memory usage and leakage by BrowserUk (Patriarch) on Jun 23, 2011 at 15:58 UTC
Re^2: memory usage and leakage by szabgab (Priest) on Jun 23, 2011 at 11:10 UTC
Thanks, I especially liked your sig :)	[reply]
Re: memory usage and leakage by mirod (Canon) on Jun 23, 2011 at 12:46 UTC
when running on SuSE with perl 5.10.0 I get the following data: Did you try using a different version of Perl? I don't follow p5p but I know by experience that 5.10.0 had some memory leaks that were fixed in 5.10.1 and above. perldelta for 5.10.1 in particular lists a weak reference to a hash would leak.	[reply]
Re^2: memory usage and leakage by szabgab (Priest) on Jun 23, 2011 at 13:28 UTC
Good idea. I have not tried yet as our production perl is 5.10.0 and I cannot change that any time soon. Besides it seems the leak is in one of the modules I am using.	[reply]
Re^3: memory usage and leakage by mirod (Canon) on Jun 23, 2011 at 13:31 UTC
The leak in Perl can show up in one of the modules. It happened to XML::Twig, I had a couple of reports that it was leaking, and when I looked into it, it turned out to be a Perl problem. updating Perl to 5.10.1 fixed the problem for everyone.	[reply]
Re^3: memory usage and leakage by Anonymous Monk on Jun 23, 2011 at 13:50 UTC
Load perldeltas - every perl*delta in one file, and search (Ctrl+F) for leak, its amazing what you can find :) http://pastehtml.com/view/1e8ud19.html#p5140-Miscellaneous_Memory_Leaks http://pastehtml.com/view/1e8ud19.html#p5101-Selected_Bug_Fixes	[reply]
memory leakage by szabgab (Priest) on Jun 23, 2011 at 12:14 UTC
I went further and tried the Net::XMPP code: #!/usr/bin/perl use strict; use warnings; print_size('empty'); use Net::XMPP; print_size("after Net::XMPP"); for (1..10) { my $conn = Net::XMPP::Client->new; print_size('Net::XMPP::Client created'); my $status = $conn->Connect( hostname => 'talk.google.com', port => 5222, componentname => 'gmail.com', connectiontype => 'tcpip', tls => 1, ); print_size('connection started'); $conn = undef; } sub print_size { my ($msg) = @_; my @lines = qx{/bin/ps -e -o pid,ppid,vsize,rss,command \| grep ^$$ +}; chomp @lines; foreach my $line (@lines) { my ($pid, $ppid, $vsize, $rss) = split /\s+/, $line; print "VM: $vsize RSS: $rss - $msg\n"; } return; } [download] The result is VM: 48596 RSS: 14648 - empty VM: 48596 RSS: 14652 - after Net::XMPP VM: 48808 RSS: 14744 - Net::XMPP::Client created VM: 67920 RSS: 18344 - connection started VM: 67920 RSS: 18344 - Net::XMPP::Client created VM: 68072 RSS: 18408 - connection started VM: 68072 RSS: 18408 - Net::XMPP::Client created VM: 68192 RSS: 18464 - connection started VM: 68192 RSS: 18464 - Net::XMPP::Client created VM: 68192 RSS: 18516 - connection started VM: 68192 RSS: 18516 - Net::XMPP::Client created VM: 68300 RSS: 18572 - connection started VM: 68300 RSS: 18572 - Net::XMPP::Client created VM: 68412 RSS: 18648 - connection started VM: 68412 RSS: 18648 - Net::XMPP::Client created VM: 68516 RSS: 18712 - connection started VM: 68516 RSS: 18712 - Net::XMPP::Client created VM: 68636 RSS: 18784 - connection started VM: 68636 RSS: 18784 - Net::XMPP::Client created VM: 68740 RSS: 18856 - connection started VM: 68740 RSS: 18856 - Net::XMPP::Client created VM: 68848 RSS: 18928 - connection started [download] So if I understand correctly this means the Net::XMPP::Client is leaking memory? How could I better check it?	[reply] [d/l] [select]
Re: memory leakage by szabgab (Priest) on Jun 23, 2011 at 12:33 UTC
Added `use Devel::LeakGuard::Object qw(leakguard);` [download] and then leakguard around the for loop and I got: `Object leaks found: Class Before After Delta FileHandle 0 10 10 IO::Select 0 20 20 IO::Socket::SSL 0 10 10 IO::Socket::SSL::SSL_Context 0 10 10 IO::Socket::SSL::SSL_HANDLE 0 10 10 Net::XMPP::Client 0 10 10 Net::XMPP::Debug 0 10 10 XML::Stream 0 10 10 XML::Stream::Parser 0 20 20 utf8 0 1 1` [download] I guess this means there is a memory leak.	[reply] [d/l] [select]
Re: memory leakage by Anonymous Monk on Jun 23, 2011 at 12:33 UTC
So if I understand correctly this means the Net::XMPP::Client is leaking memory? How could I better check it? Well, Net::XMPP::Client or one of the modules it uses -- to narrow it down you might try Devel::Cycle, Devel::Leak, Devel::MemoryTrace::Light, Devel::LeakGuard::Object, Devel::Monitor, but its a lot of T.I.T.S. :)	[reply]
Re: memory usage and leakage by sundialsvc4 (Abbot) on Jun 23, 2011 at 13:36 UTC
A few things to think about: Do you have a memory leak, or do you simply have memory consumption? Test::Memory::Cycle is a good way to determine if you have self-referential data structures in a program. (See also the other CPAN modules which it uses.) Is there memory pressure? (If there isn’t, then the point-of-view of the always lazy operating system will be, “always room for one more!” And, it is absolutely correct.) The metrics produced by commands such as `ps` are only the roughest of indicators. What you are really interested in is paging, particularly page steals, which is the actual indicator that something actually hurts. If you don’t have pain, you don’t have a problem. Don’t guess ... measure. If you suspect that excessive memory consumption is the root cause of your problem, demonstrate it. “Chips are cheap” now, whereas time is as expensive as ever. I’m not talking about computer time here ... I’m talking about your own. Sometimes the very best way to construct a program is to use the strongest software tools you can, then deal with any memory issues that actually develop (don’t borrow trouble ...) by “throwing silicon at it.” After all, it’s only sand ...	[reply]
Re^2: memory usage and leakage by Sewi (Friar) on Jun 23, 2011 at 15:24 UTC
For 2 & 4: You're right - for web or short term applications. Net::XMPP is used for IM connections and those are usually long-life. If there is any real memory leak (which increases the used memory over time even without additional data being stored), it will become a problem, sooner or later. Let the final script run for a hour or day and everything might be fine, but if it's logged into XMPP for weeks or even longer, memory usage will grow endless. The problem is even worse, because there are running multiple instances (dosn't matter if they are processes or threads). Restarting the whole application from time to time is bad, even if the new process re-logins to XMPP after some seconds. The problem will catch you sooner or later even if you try to reboot the app depending on memory usage. Try Padre - the free Perl IDE	[reply]
Re^2: memory usage and leakage by szabgab (Priest) on Jun 26, 2011 at 10:32 UTC
1. Sure, there was a leak. I could see it even without Test::Memory::Cycle but using that module helped me locate it and fix it. Thanks for pointing in it out. 2. Sure there is a pressure. We made some simple calculations of how much memory we will need and it it was more than what the servers can handle. 3. Wasn't my question exactly about how to measure?	[reply]
Re: memory usage and leakage by Anonymous Monk on Jun 23, 2011 at 11:41 UTC
Here is a win32 version, perl v5.12.2 Read more... (2 kB) VM: 18660 < WS: 2756 VM: 608 > empty VM: 18660 < WS: 2956 VM: 636 > strict VM: 18660 < WS: 3092 VM: 760 > warnings VM: 18660 < WS: 3424 VM: 1024 > Module::Load VM: 18660 < WS: 3644 VM: 1216 > Module::Loaded VM: 20788 < WS: 4520 VM: 2036 > Module::Pluggable VM: 20972 < WS: 5084 VM: 2504 > Time::HiRes VM: 20972 < WS: 5168 VM: 2560 > Time::Local VM: 21020 < WS: 5556 VM: 2928 > Time::Piece VM: 21020 < WS: 5556 VM: 2928 > Time::Seconds VM: 21020 < WS: 5796 VM: 3172 > Time::gmtime VM: 21020 < WS: 5816 VM: 3184 > Time::localtime VM: 21020 < WS: 5816 VM: 3184 > Time::tm VM: 25152 < WS: 6544 VM: 3896 > Pod::Perldoc VM: 25152 < WS: 7660 VM: 4996 > Pod::Simple VM: 25220 < WS: 7996 VM: 5324 > Data::Dumper VM: 25220 < WS: 7996 VM: 5324 > Scalar::Util VM: 25220 < WS: 8404 VM: 5728 > Getopt::Long VM: 25352 < WS: 10188 VM: 7444 > LWP::UserAgent VM: 34232 < WS: 17548 VM: 14592 > Moose VM: 52272 < WS: 22800 VM: 19232 > POE => qw(Component::Server::TC +P) [download]	[reply] [d/l] [select]


Pathologically Eclectic Rubbish Lister
	PerlMonks