Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Perl Memory problem ...

by Sascha2018 (Acolyte)
on May 09, 2020 at 20:16 UTC ( [id://11116619]=perlquestion: print w/replies, xml ) Need Help??

Sascha2018 has asked for the wisdom of the Perl Monks concerning the following question:

Hello. I have programmed a Chatserver in Perl. Now if i look in my System Monitor , i can see, that the perl process needs 10 MB at start. Later, if anybody writes a chat message, the process grow up to 30 MB or more. So now my question: How can i release Memory ( RAM ) ? I tried to use the perlfunction: undef but nothing happens :( Thank you Regards Sascha

Replies are listed 'Best First'.
Re: Perl Memory problem ... (Memory Tools References)
by eyepopslikeamosquito (Archbishop) on May 10, 2020 at 05:49 UTC
[reply]
[d/l]
[select]
Re: Perl Memory problem ...
by davido (Cardinal) on May 10, 2020 at 16:04 UTC

    Perl won't release memory back to the operating system at runtime, only on global destruction as the script terminates. However, Perl will re-use the memory that it has allocated and that it is no longer using. For example, if you declare a lexical variable in one scope and load it up with 30MB of data, but then let that variable fall out of scope, Perl should be able to re-use that memory space.

    Places to look:

    • Circular references: These can cause memory leaks since the memory never has its reference count drop to zero, Perl can't re-claim it for internal use.
    • Modules: Are you using half of CPAN? There's nothing wrong with using Moose, DBIx::Class, and many of the other slightly heavy distributions on CPAN, but they do use memory.
    • Scoping: Are you allowing lexical variables to fall out of scope? (You should be.) Program with the objective of keeping state as ephemeral as possible; let variables have the narrowest possible scope - particularly those that are holding large chunks.
    • File/Database handling: Are you slurping in large files or large Database query data sets? Process smaller chunks at a time.
    • Stability: Once your process grows to 30MB, does it stay there, or does it grow further over time? Frankly, even if it's consuming 100MB, that's not the end of the world as long as you can count on it staying close to that plateau. If it keeps on growing over time, you've got a bigger problem (a memory leak). A well-behaved process that hits a memory plateau and stays there is usually acceptable.
    • Tools: Try valgrind, Devel::LeakTrace, Devel::MemoryTrace::Lite, and other such tools can help you to watch how memory is growing and being used.

    Dave

[reply]

      Perl won't release memory back to the operating system at runtime, only on global destruction as the script terminates.
      While that may be true for most practical purposes, I believe Perl simply defers to the implementation of the user-level malloc function - see for example: Memory is not released back to operating system.

      Last time I looked, both Linux and Windows malloc do return "large" chunks back to the OS.

[reply]
[d/l]
[select]

        Thanks. I was probably misremembering Tim Bunce's YAPC talk: https://www.youtube.com/watch?v=GIIeOntmojg. I'll have to review it again when I have a moment. Your research looks thorough, though.

        Update: At about 4:10 Tim discusses malloc behavior, and suggests that larger allocations are mmapped, and the gist seems to be that while rare for malloc to actually give back, it is more likely for those larger mmapped chunks. So, thanks for motivating me to find that talk again. I'll have to watch the rest of it again this evening. It does confirm your assertion, too.


        Dave

[reply]
[d/l]
Re: Perl Memory problem ...
by haukex (Archbishop) on May 09, 2020 at 22:11 UTC

    That's not much information to go on, and nowadays, 30MB isn't that much at all. Perhaps you could show some code to reproduce the issue (SSCCE, How do I post a question effectively?), and explain what the problem is? Like for example, if the memory use continually goes up, that would indicate a memory leak.

[reply]
Re: Perl Memory problem ...
by Corion (Patriarch) on May 09, 2020 at 20:49 UTC

    Somewhere, you're keeping memory alive.

    Without seeing your code, it is impossible for us to know how to change your program.

    Either reduce your code and post a short, self-contained example, or consider how you can periodically restart your chat server so all accumulated memory gets reclaimed.

[reply]
Re: Perl Memory problem ...
by perlfan (Vicar) on May 12, 2020 at 02:36 UTC
    Need to see what you're doing. You don't need to release memory to the OS if you've scoped things properly. Another suggestion, don't import everything in your use statements. Idk what you did here, but I can image a list of imported modules a mile long. That or you used Moose or Moo and still included the kitchen sink.
[reply]
[d/l]
Re: Perl Memory problem ... (circular references)
by Anonymous Monk on May 10, 2020 at 04:36 UTC
[reply]
Re: Perl Memory problem ...
by Sascha2018 (Acolyte) on May 13, 2020 at 17:17 UTC
    Hello. Thanks for the replies. Here is a little bit of my perl script i wrote. The problem is: At start time it uses 10MB Memory. And if a user writes a message ( SENDMESSAGE ) , it needs for every message ( 300 KB of memory ). Now ... how can i fix the problem? 
    use feature 'state';
use Time::HiRes;
require "config.cfg";
require "Subs.pl";
require "format.pl";
require "server/mainprogram_subs.pl";
require "ConnectDB.pl";
require "loadconfig.pl";
require "$config{language}.lng";
require 5.002;
use IO::Socket::SSL 'inet4';
$IO::Socket::SSL::DEBUG = 3;
use IO::Select;
use Time::HiRes;
use HTML::Template;
use  Text::Wrap;
use Fcntl ':flock';
use strict;
no strict 'refs';

    [download]
    later in my program: 
    if($buffer =~ /^SENDMESSAGE$digest\|(.*?)$/){
                          my $what = $1;
                          my($room,$touser,$kind,$message,$fromuser)= 
+split /\|/, $what;
                          my $myuser = $fromuser;
                          my $update_sock = getCurrentIdent($myuser);
                          if( SockExists($update_sock) ) {
                            $clients{$update_sock}->{lastsrvpost} = ti
+me;
                          }
                          if($touser eq "" && $room ne "" && $kind ne 
+"PRVMESSAGE")    {  sendToSocket($message, { toroom => $room, kind =>
+ $kind, fromuser => $myuser  } );
                          }elsif($touser ne "" && $room eq "" && $kind
+ ne "PRVMESSAGE" ) { sendToSocket($message,{ fromuser => $myuser, tou
+ser => $touser, kind => $kind } );
                          }elsif($touser eq "" && $room eq "" && $kind
+ ne "PRVMESSAGE") { sendToSocket($message, {});
                          }
                          next;

    [download]
    and the sendToSocket routine is the following: 
    sub sendToSocket {
    my( $data, $params ) = @_;
    if( $data eq "" ) {
      return 0;
   }
    if( $params->{toroom} ne "" && $params->{touser} eq "" ) {
             foreach my $ident( keys %clients ) {
                 our $nick = $clients{$ident}->{nick};
                 if( isInIgnorelist($nick, $params->{fromuser}) && $co
+nfig{ignore_modus}{$params->{kind}}==1) {
               }else{
                    my %info = loadMemberInfo($nick);
                    if( lc $params->{toroom} eq lc $info{room}) {
                       my $fh = $clients{$ident}->{sock};
                       if( defined $fh ){
                       my $msg = createFilter($ident, $data);
                       $fh->syswrite($msg . "\n");
                       undef $fh;
                       undef $msg;
                       undef %info;
                       undef $nick;
                       }
                    }
               }
              }
    }elsif( $params->{toroom} eq "" && $params->{touser} ne "" ) {
              my $user = $params->{touser};
              my $sock = getCurrentSocket($user);
              my $sn= $sock->{sockname};
              my $socket = $sock->{socket};
              my $ident = getCurrentIdent($user);
              if( isInIgnorelist($clients{$sn}->{nick}, $params->{from
+user}) && $config{ignore_modus}{$params->{kind}}==1 ) {
              }else{
                 if( defined $socket ){
                   $msg = createFilter($ident, $data);
                   $socket->syswrite($msg . "\n");
                   undef $msg;
                   undef $ident;
                   undef $sock;
                 }
              }
    }elsif( $params->{touser} eq "" && $params->{toroom} eq "" ) {
              foreach my $ident( keys %clients ) {
                  my $sock = $clients{$ident}->{sock};
                  $msg = createFilter($ident, $data);
                  if( defined $sock ){
                  $sock->syswrite($msg . "\n");
                  }
                  undef $msg;
                  undef $sock;
              }
    }
}

    [download]
    But if i use undef, nothing happens. It still uses the memory :( Hope you can help. Thanks Regards Sascha
[reply]
[d/l]
[select]

      Most likely, your problem lies somewhere else as here you clean out the variables you create locally.

      Take a look at Devel::Cycle and/or Devel::LeakTrace and Devel::FindRef to see where your program is collecting memory. Ideally, you can eliminate all server/client interaction and use a faked client message in a loop to reproduce the problem.

      You might want to change the coding style from

                       our $nick = $clients{$ident}->{nick};

      [download]

      ... to

                       my $nick = $clients{$ident}->{nick};

      [download]
[reply]
[d/l]
[select]
Re: Perl Memory problem ...
by Anonymous Monk on May 11, 2020 at 03:09 UTC
    Remember that operating systems are designed to be lazy. If there's no pressure on the memory resource, it really doesn't care to reduce the exact amount that you are now using. Should pressure later develop, it will try more aggressively to determine which portions of your memory are now being used and which can be safely stolen for other purposes. But, until that actually occurs, it won't bother.
[reply]
A reply falls below the community's threshold of quality. You may see it by logging in.

Back to Seekers of Perl Wisdom

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?

www.com | www.net | www.org
Node Status?
node history
Node Type: perlquestion [id://11116619]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? • Last hour • Other CB clients
Other Users?
Others surveying the Monastery: (2)
As of 2024-04-25 22:44 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found