Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Building a small perl

by awy (Acolyte)
on Mar 20, 2009 at 12:01 UTC ( [id://752003]=perlquestion: print w/replies, xml ) Need Help??

awy has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to build a version of perl 5.10 that will have as small a memory footprint (both code and data) as possible. I want to run an existing perl application with a small memory budget. The application uses quite a bit of perl and CPAN stuff and typically has a memory footprint in the range 60-120MB on common platforms.

So far I have built perl with the following (possibly relevant) configure arguments:

config_arg18='-Ubincompat5005' config_arg19='-Ud_dosuid' config_arg20='-Ui_db' config_arg21='-Ui_gdbm' config_arg22='-Ui_ndbm' config_arg23='-Uuse64bitall' config_arg24='-Uuse64bitint' config_arg25='-Uuseithreads' config_arg26='-Uuselargefiles' config_arg27='-Uuselongdouble' config_arg28='-Uusemorebits' config_arg29='-Uuseshrplib' config_arg30='-Uusethreads'
Are there any other simple suggestions?

Or not so simple ones? I'd be prepared to go as far as modifying perl internals if it is likely to be useful.

I found the following: oplines - make perl runtime 8% faster and smaller; Does anyone know if this made it into 5.10? I would guess not, given the date of the messages.

Finally (for now) are there any generic tips for perl code to reduce the size of the in-memory (compiled) size? These could be general patterns, or processing techniques that one might apply when building the deployable application.

Replies are listed 'Best First'.
Re: Building a small perl
by zentara (Archbishop) on Mar 20, 2009 at 14:48 UTC
    If you can give up some functionality, there is "microperl". Google for microperl, and read the Install instructions.....search the Perl source distribution for "microperl", there are detailed steps to make it. There is (was) also miniperl, that had a bit more functionality.

    Actually I was thinking that microperl could be a basis for a limited web-browser plugin.


    I'm not really a human, but I play one on earth My Petition to the Great Cosmic Conciousness
      Thanks. I don't think microperl, or even miniperl, is going to cut it. As I said, we use a lot of CPAN code, and that includes plenty of XS modules. Also, to answer an early question, there is actually enough code for that to be an issue: > 45MB.
Re: Building a small perl
by jethro (Monsignor) on Mar 20, 2009 at 13:01 UTC

    Depends on your application. I would guess that (code) size of your own code should not be a problem (if you haven't developed it for years). It is either your data or the combined size of many CPAN-modules. Your own data you can minimize by removing global vars (if any), not having big arrays in memory, things you probably thought of yourself already.

    CPAN modules are probably not built with memory efficiency in mind. You might be able to switch out modules with comparable functionality but smaller footprint or more importantly a smaller list of dependencies (or even replace them with your own code). But for that to work you should know the size of the cpan-modules you use. Something like perl -e 'use somemodule; while (1) {}' to check the size with 'ps' or 'top' might be sufficient to give you an idea of the base memory need of a module with dependencies. Better might be some test code with a typical use case

    Some modules might have reserved space for caches. Check their documentation or their code

Re: Building a small perl
by sundialsvc4 (Abbot) on Mar 20, 2009 at 12:42 UTC

    You may be following the same primrose path that I just went down. The bottom line was, no shared-host shop that my client had ever heard of (I know that there are others who specialize in Perl...) would deploy Apache with quite... enough memory to do the job. I had to switch to a dedicated-host.

    I rather doubt that your experience would be much different. If resources are a problem, it's undoubtedly some artificial limit that just has to be bypassed. Otherwise you drop fairly-immediately into the land of “diminishing returns...” where, even if you could get it to work, it's not worth the effort.

Re: Building a small perl
by samtregar (Abbot) on Mar 20, 2009 at 21:40 UTC
    I'm curious, did any of that help? I can't imagine it shaved more than a few hundred k. And a lot of that "saving" would be in shared libs that are probably already in memory.

    You're not giving us a lot of details to work with here, but here's a few random suggestions. First, turn off caching - lots of modules support caching data and you don't want them to do that if memory is scarce. Second, switch to straight CGI if you're using mod_perl - mod_perl is fast because it keeps your app in memory, but you don't want that. CGI will immediately release all memory used by a script as soon as it returns results. Finally, look critically at which modules are being used and make sure you really need them. Having a Project::All module with use statements for all your CPAN modules is very convenient but it will kill your memory usage if you're running a script that only needs a few deps.

    If you want more useful advice you'll have to give us more information - what does the app do, what CPAN modules does it use, does it leak memory, etc.

    -sam

      I already have some reasonably good data about the size of the complied perl code (not the native executable) and how much of it is ours and how much comes from CPAN, etc.; using Devel::Size & Devel::Size::Report. With this analysis it is clear that most of this is our own code. And I really am talking about the size of the compiled code, not data.

      Actually, changing the build options did make a big difference. I think excluding threading was the big win. Altogether those options, coupled with a switch to 5.10 from 5.8, saved about 30%. But even as it stands, the in-memory image include a lot of overhead just for the filenames and module names: about one instance of each of those strings for each executable line of code in the source files.

        I think excluding threading was the big win

        Yes, you want to stick with that one.

        a switch to 5.10 from 5.8, saved about 30%

        I think a switch back to 5.6 will also provide a further significant reduction (untested). You could even go further back, if you like - though the further back you go the more likely it becomes that you'll have to make amendments to the code you want to run.

        Cheers,
        Rob
Re: Building a small perl
by Animator (Hermit) on Mar 20, 2009 at 17:55 UTC

    First: how much memory does the program use when you just run a syntax check? Or when add a exit; as the very first line of the script? (well the second after a sleep)

    Is it the compiled program that is using the memory or is it the code that is being executed at runtime that is using a lot of memory?

    What you should be doing (IMHO) is figure out what is using the memory.

    One approach could be to add a system("ps -uh $$"); after every line.

    IMHO you will not get a serious memory reduction by recompiling perl and setting/unsetting some options. The only way to reduce the memory is to look at the code of the application.

    One example to show what I mean:

    perl -wle 'system("ps uch $$");@x=1..1_000_000;print "Elements in \@x: + " . @x;; system("ps uch $$");' xxx 2395 0.0 1.0 46488 44384 pts/0 R+ 18:46 0:00 perl Elements in @x: 1000000 xxx 2395 0.0 1.5 66104 64012 pts/0 R+ 18:46 0:00 perl
    vs
    perl -wle 'system("ps uch $$");push @x, $i while ($i++ < 1_000_000);pr +int "Elements in \@x: " . @x;; system("ps uch $$");' xxx 2365 0.0 0.0 3152 1248 pts/0 R+ 18:45 0:00 perl Elements in @x: 1000000 xxx 2365 20.0 0.5 22992 21056 pts/0 R+ 18:45 0:00 perl

    The array contains exactly the same information. The first program uses 64MB of memory and the second 22MB of memory. The first one uses more memory, the second one uses more CPU.

    Same example but with a syntax check:

    perl -wlce 'system("ps uch $$");@x=1..1_000_000;print "Elements in \@x +: " . @x;; system("ps uch $$");CHECK { system("ps uch $$"); }' xxx 2554 0.0 1.0 46488 44404 pts/0 R+ 18:50 0:00 perl -e syntax OK
    vs
    perl -wlce 'system("ps uch $$");push @x, $i while ($i++ < 1_000_000);p +rint "Elements in \@x: " . @x;; system("ps uch $$");CHECK { system("p +s uch $$") }' xxx 2521 0.0 0.0 3152 1260 pts/0 R+ 18:49 0:00 perl -e syntax OK

    What this shows is that in the first program 45MB of memory is already being used before the code is being executed. That memory will only be free'ed/reused when perl stops.

    update: To provider an anwsers on your question about: 'oplines - make perl runtime 8% faster and smaller' that messages is theoretical and you such read it as such. It certainly will not make the runtime of each and every application 8% faster and smaller. It all depends on where the memory is being used.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://752003]
Approved by Corion
Front-paged by Arunbear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others imbibing at the Monastery: (3)
As of 2024-03-29 07:01 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found