szabgab has asked for the wisdom of the Perl Monks concerning the following question:
An application I am writing uses a lot of memory and there seem to be a serious memory leakage I try to track down.
As I am far from being a memory expert I wonder if even I am looking at the right numbers and if the baseline information is correct?
I am using the following script to check its own memory usage:
#!/usr/bin/perl
use strict;
use warnings;
print_size('empty');
foreach my $module (
'Data::Dumper',
'Scalar::Util',
'Getopt::Long',
'Net::XMPP',
'JSON::XS',
'LWP::UserAgent',
'POE qw(Component::Server::TCP)',
'Moose',
) {
eval "use $module";
die $@ if $@;
print_size("after $module");
}
sub print_size {
my ($msg) = @_;
my @lines = qx{/bin/ps -e -o pid,ppid,vsize,rss,command | grep ^$$
+};
chomp @lines;
foreach my $line (@lines) {
my ($pid, $ppid, $vsize, $rss) = split /\s+/, $line;
print "VM: $vsize RSS: $rss - $msg\n";
}
return;
}
and when running on SuSE with perl 5.10.0
I get the following data:
VM: 15972 RSS: 2324 - empty
VM: 18740 RSS: 3256 - after Data::Dumper
VM: 20968 RSS: 3388 - after Scalar::Util
VM: 21760 RSS: 4240 - after Getopt::Long
VM: 51348 RSS: 15408 - after Net::XMPP
VM: 53648 RSS: 15588 - after JSON::XS
VM: 57980 RSS: 17704 - after LWP::UserAgent
VM: 65852 RSS: 21572 - after POE qw(Component::Server::TCP)
VM: 88616 RSS: 31924 - after Moose
So if I understand correctly my script starts - after loading all the necessary modules - with 88 Mb memory used?
Do I see correctly that loading Moose take 20Mb without even creating classes and objects?
Net::XMPP is even bigger as it takes 30Mb ?
Is the way I am measuring correct?
As I'll need to run several instances of this process at the same time will they share some memory or will this number just be multiplied by the number of processes I have?
Re: memory usage and leakage
by Sewi (Friar) on Jun 23, 2011 at 11:06 UTC
|
The VM value is the address space allocated by an application, not the memory used. It doesn't give any reasonable result at all.
RSS is the memory actually used and this is not shared across instances. The sum of RSS of all processes is the memory (including swap) used on your computer. Add kernel memory, disk cache and free and you got all of your memory.
If you want to share some memory, "use" all required modules and start threading, but you need to check if all modules are thread-safe and ensure your code to be thread-safe.
If you care about CPU or memory usage, remove PEO and Moose from your project as they eat up half of the scripts memory.
| [reply] |
|
If you want to share some memory, "use" all required modules and start threading,
Unless the application requires all the modules in all the threads, that is bad advice. It ensures that every thread will get it own copy of everything in every module, whether it ever uses it or not. And if the application does require all of those in every thread, it is badly designed and probably won't work anyway.
If the OP structures his application properly, and requires only those modules he needs in each thread, it will reduce memory consumption conspicuously.
But you're right that the simplest route to reducing memory consumption is to avoid behemoth frameworks that force you to load everything before you can do the simplest thing, and a shit-load more every time you want to do something extra.
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply] |
|
You're right as long as the threads do different things.
But he said, he'd need many instances, my suggestion was to make every "instance" one "thread".
| [reply] |
|
|
Thanks, I especially liked your sig :)
| [reply] |
Re: memory usage and leakage
by mirod (Canon) on Jun 23, 2011 at 12:46 UTC
|
when running on SuSE with perl 5.10.0 I get the following data:
Did you try using a different version of Perl? I don't follow p5p but I know by experience that 5.10.0 had some memory leaks that were fixed in 5.10.1 and above.
perldelta for 5.10.1 in particular lists a weak reference to a hash would leak.
| [reply] |
|
Good idea.
I have not tried yet as our production perl is 5.10.0 and I cannot change that any time soon.
Besides it seems the leak is in one of the modules I am using.
| [reply] |
|
The leak in Perl can show up in one of the modules. It happened to XML::Twig, I had a couple of reports that it was leaking, and when I looked into it, it turned out to be a Perl problem. updating Perl to 5.10.1 fixed the problem for everyone.
| [reply] |
|
| [reply] |
memory leakage
by szabgab (Priest) on Jun 23, 2011 at 12:14 UTC
|
I went further and tried the Net::XMPP code:
#!/usr/bin/perl
use strict;
use warnings;
print_size('empty');
use Net::XMPP;
print_size("after Net::XMPP");
for (1..10) {
my $conn = Net::XMPP::Client->new;
print_size('Net::XMPP::Client created');
my $status = $conn->Connect(
hostname => 'talk.google.com',
port => 5222,
componentname => 'gmail.com',
connectiontype => 'tcpip',
tls => 1,
);
print_size('connection started');
$conn = undef;
}
sub print_size {
my ($msg) = @_;
my @lines = qx{/bin/ps -e -o pid,ppid,vsize,rss,command | grep ^$$
+};
chomp @lines;
foreach my $line (@lines) {
my ($pid, $ppid, $vsize, $rss) = split /\s+/, $line;
print "VM: $vsize RSS: $rss - $msg\n";
}
return;
}
The result is
VM: 48596 RSS: 14648 - empty
VM: 48596 RSS: 14652 - after Net::XMPP
VM: 48808 RSS: 14744 - Net::XMPP::Client created
VM: 67920 RSS: 18344 - connection started
VM: 67920 RSS: 18344 - Net::XMPP::Client created
VM: 68072 RSS: 18408 - connection started
VM: 68072 RSS: 18408 - Net::XMPP::Client created
VM: 68192 RSS: 18464 - connection started
VM: 68192 RSS: 18464 - Net::XMPP::Client created
VM: 68192 RSS: 18516 - connection started
VM: 68192 RSS: 18516 - Net::XMPP::Client created
VM: 68300 RSS: 18572 - connection started
VM: 68300 RSS: 18572 - Net::XMPP::Client created
VM: 68412 RSS: 18648 - connection started
VM: 68412 RSS: 18648 - Net::XMPP::Client created
VM: 68516 RSS: 18712 - connection started
VM: 68516 RSS: 18712 - Net::XMPP::Client created
VM: 68636 RSS: 18784 - connection started
VM: 68636 RSS: 18784 - Net::XMPP::Client created
VM: 68740 RSS: 18856 - connection started
VM: 68740 RSS: 18856 - Net::XMPP::Client created
VM: 68848 RSS: 18928 - connection started
So if I understand correctly this means the Net::XMPP::Client is leaking memory? How could I better check it?
| [reply] [d/l] [select] |
|
use Devel::LeakGuard::Object qw(leakguard);
and then leakguard around the for loop and I got:
Object leaks found:
Class Before After Delta
FileHandle 0 10 10
IO::Select 0 20 20
IO::Socket::SSL 0 10 10
IO::Socket::SSL::SSL_Context 0 10 10
IO::Socket::SSL::SSL_HANDLE 0 10 10
Net::XMPP::Client 0 10 10
Net::XMPP::Debug 0 10 10
XML::Stream 0 10 10
XML::Stream::Parser 0 20 20
utf8 0 1 1
I guess this means there is a memory leak. | [reply] [d/l] [select] |
|
| [reply] |
Re: memory usage and leakage
by sundialsvc4 (Abbot) on Jun 23, 2011 at 13:36 UTC
|
| [reply] |
|
For 2 & 4:
You're right - for web or short term applications. Net::XMPP is used for IM connections and those are usually long-life. If there is any real memory leak (which increases the used memory over time even without additional data being stored), it will become a problem, sooner or later.
Let the final script run for a hour or day and everything might be fine, but if it's logged into XMPP for weeks or even longer, memory usage will grow endless.
The problem is even worse, because there are running multiple instances (dosn't matter if they are processes or threads).
Restarting the whole application from time to time is bad, even if the new process re-logins to XMPP after some seconds. The problem will catch you sooner or later even if you try to reboot the app depending on memory usage.
| [reply] |
|
| [reply] |
Re: memory usage and leakage
by Anonymous Monk on Jun 23, 2011 at 11:41 UTC
|
Here is a win32 version, perl v5.12.2
VM: 18660 < WS: 2756 VM: 608 > empty
VM: 18660 < WS: 2956 VM: 636 > strict
VM: 18660 < WS: 3092 VM: 760 > warnings
VM: 18660 < WS: 3424 VM: 1024 > Module::Load
VM: 18660 < WS: 3644 VM: 1216 > Module::Loaded
VM: 20788 < WS: 4520 VM: 2036 > Module::Pluggable
VM: 20972 < WS: 5084 VM: 2504 > Time::HiRes
VM: 20972 < WS: 5168 VM: 2560 > Time::Local
VM: 21020 < WS: 5556 VM: 2928 > Time::Piece
VM: 21020 < WS: 5556 VM: 2928 > Time::Seconds
VM: 21020 < WS: 5796 VM: 3172 > Time::gmtime
VM: 21020 < WS: 5816 VM: 3184 > Time::localtime
VM: 21020 < WS: 5816 VM: 3184 > Time::tm
VM: 25152 < WS: 6544 VM: 3896 > Pod::Perldoc
VM: 25152 < WS: 7660 VM: 4996 > Pod::Simple
VM: 25220 < WS: 7996 VM: 5324 > Data::Dumper
VM: 25220 < WS: 7996 VM: 5324 > Scalar::Util
VM: 25220 < WS: 8404 VM: 5728 > Getopt::Long
VM: 25352 < WS: 10188 VM: 7444 > LWP::UserAgent
VM: 34232 < WS: 17548 VM: 14592 > Moose
VM: 52272 < WS: 22800 VM: 19232 > POE => qw(Component::Server::TC
+P)
| [reply] [d/l] [select] |
|
|