Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

slow startup for some common modules? (autodie, Pod::Usage, Getopt::Long))

by almr (Sexton)
on Dec 27, 2022 at 22:05 UTC ( [id://11149146]=perlquestion: print w/replies, xml ) Need Help??

almr has asked for the wisdom of the Perl Monks concerning the following question:

I've been working on a "quick and dirty" script that should run from .bashrc. It seemed a lot slower than an equivalent bash script (which runs several executables...), so I decided to time it. It turned out the code itself didn't really make a difference -- it was the imports that were sluggish.

In particular, just importing (without using at all) autodie, Pod::Usage and Getopt::Long will slow down a do-nothing program. Here are the results from hyperfine (-adie means enable autodie, -podu = Pod::Usage, and -go = Getopt::Long; actual programs at the end)

Benchmark 1: /tmp/x-go-adie-podu.pl Time (mean ± s): 107.1 ms ± 0.5 ms [User: 94.5 ms, Sys: 12.5 ms] Range (min … max): 106.2 ms … 108.1 ms 27 runs Benchmark 2: /tmp/x-go-adie.pl Time (mean ± s): 50.2 ms ± 0.2 ms [User: 44.5 ms, Sys: 5.7 ms] Range (min … max): 49.7 ms … 50.6 ms 58 runs Benchmark 3: /tmp/x-go.pl Time (mean ± s): 25.7 ms ± 0.1 ms [User: 21.6 ms, Sys: 4.1 ms] Range (min … max): 25.5 ms … 25.9 ms 112 runs Benchmark 4: /tmp/x.pl Time (mean ± s): 7.8 ms ± 0.1 ms [User: 5.3 ms, Sys: 2.5 ms] Range (min … max): 7.6 ms … 8.1 ms 324 runs

So, we jump from 8ms to 25ms to 50ms to 107ms just by importing (in order) Getopt::Long, autodie, and Pod::Usage. Once it reaches tenths-of-a-second territory I get itchy, because it's starting to become visible to humans.

Anyway

  • why are these so slow to import?
  • can they be easily replaced? I could ditch autodie; I could use perldoc to pre-generate the usage message; but I'd rather not hand-code an arg parsing loop.

Actual files:

### /tmp/x-go-adie-podu.pl #!/usr/bin/env perl use strict; use warnings; use Getopt::Long qw( :config ); use Pod::Usage; use autodie; ### /tmp/x-go-adie.pl #!/usr/bin/env perl use strict; use warnings; use Getopt::Long qw( :config ); use autodie; ### /tmp/x-go.pl #!/usr/bin/env perl use strict; use warnings; use Getopt::Long qw( :config ); ### /tmp/x.pl #!/usr/bin/env perl use strict; use warnings;

Replies are listed 'Best First'.
Re: slow startup for some common modules? (autodie, Pod::Usage, Getopt::Long))
by hippo (Bishop) on Dec 28, 2022 at 10:15 UTC
    why are these so slow to import?

    Well, if you think about it when you use Pod::Usage it has to parse the source of your code, extract the POD and process the POD to present usage info - that will all take time. You can save this by postponing the load until/unless it is actually needed (most runs will never need it, of course). The other option is to use pod2usage out-of-band in advance and just copy the output of that into your code instead statically.

    I find that Getopt::Std is much faster than Getopt::Long. Its reduced functionality may well be worth the trade-off for you.


    🦛

      By "postponing until needed", do you mean something like eval q{use Pod::Usage}, whenever the help message is needed ?

      Thanks for the pointer to Getopt::Std, I'll consider it the next time around. I've ended up coding a simple arg-parsing loop in the meanwhile (the repo is ssh_agent_share; quite trivial, but it seems I end up learning something new every time I reach back to perl).

        use Pod::Usage; # is the same as BEGIN { require Pod::Usage; Pod::Usage->import(); }
        BEGIN blocks execute at compile time, not run time.

        Assuming that often you run without ever using the Pod functions, and you want to fire that module up only if you will be needing it, you could have a runtime flag like this:

        if ($needPod) { require Pod::Usage; Pod::Usage->import(); }
        I leave it you to decide how this applies to your code. You have to do the require and import before using any functionality of Pod::Usage.

        Usually GetOPt::Std is enough for me. I almost never use autodie and absolutely never use it in end user code.

        I use this recipe in most scripts:
        use Getopt::Long; sub pod2usage { require Pod::Usage; goto \&Pod::Usage::pod2usage; } GetOptions( ... ... ) or pod2usage(2);
        I don't mind paying for Getopt::Long because I like its features, but there's no reason to load Pod::Usage unless the user screws up the commandline.

        On that note, someone ought to contribute a patch to Pod::Usage that avoids loading anything until it gets called. Then I could skip that bit of ugly boilerplate in my code.

        The eval q{use Pod::Usage}; will work, though those who are allergic to stringy evals will probably prefer something more like

        require Pod::Usage;
        Pod::Usage::pod2usage( ... );
        
Re: slow startup for some common modules? (autodie, Pod::Usage, Getopt::Long))
by LanX (Saint) on Dec 27, 2022 at 22:22 UTC
    Perl startup is "slow" by some measures.

    It starts the engine, does a lot of initializing, has to load various modules from filesystem and if they are lots of pure Perl need to compile them, and probably need to repeat all this to import more dependencies.

    So many things to consider, such that your benchmarks should be more detailed, probably using the NYT profiler.

    I don't think anything below 0.3 secs is really worth worrying, but if you really need to run a program many times you should consider talking to a server with everything pre-initialized. That's the trick behind fast-cgi etc... You'll trade space for time like that.

    But if that's not an option because you have too many different programs which need boosting, you could consider bundling all dependency into one big package with par-packer and precompiling them into a .plc with bytecode and store them in a ram-disk.

    Disclaimer: Never tried that myself, cause as I said, Perl startup is usually fast enough for me. (perltidy probably being the only exception)

    Cheers Rolf
    (addicted to the 𐍀𐌴𐍂𐌻 Programming Language :)
    Wikisyntax for the Monastery

      Well, as I've noticed many times before, and as the hyperfine above shows, the actual startup cost of perl itself is not bad at all (a few msec). For me it's only these three modules that really bite (in particular Pod::Usage, unused, takes 50msec). I had no problem with, for example, Time::HiRes, Fcntl, or Sys::Hostname. In my case, I think it makes sense to replace the slow modules (though I'm still surprised they are that slow). I've tried pp to create an executable (which seems to be a self-extracting zip) and it ran about three times slower.
Re: slow startup for some common modules? (autodie, Pod::Usage, Getopt::Long))
by kcott (Archbishop) on Dec 29, 2022 at 02:55 UTC

    G'day almr,

    I made verbatim copies of your four scripts and ran them using time (I don't have hyperfine). I performed multiple runs on each; I got results that were of the same order of magnitude as yours. So, there's nothing problematic with your system in this regard.

    I see some workarounds have been suggested. I use the three modules you tested in most of my $work scripts; I've never had any comments about slowness. I don't imagine any of your users will complain that a script took 100ms to start running. :-)

    You'll no doubt be loading additional modules which will increase the startup time. Keep an eye on what you're importing.

    # Import nothing -- fastest but requires additional coding use Some::Module (); ... Some::Module::some_function(); Some::Module::other_function(); # Import just the parts you intend to use -- still fast; less coding use Some::Module qw{some_function other_function}; ... some_function(); other_function(); # Import everything -- could be very slow use Some::Module; ... any_function();

    If you're really concerned about startup speeds, put something like this at the start of your script:

    BEGIN { print "Loading app (could take a second or two) ...\n"; }

    Then, when it only takes 100ms or so, your users should be very pleased.

    — Ken

      Thanks for taking a look. For a standalone "application", 100msec for sure doesn't matter. However, if (1) I'm calling several perl scripts inside bashrc, or (2) a script is invoked after every command (PROMPT_COMMAND in bash) or (3) a script runs in an editor on every keystroke (see the vim-like editor Kakoune), those 100msec add up. 10 scripts taking 100msec each result in a 1sec delay before being able to type anything, which is of course unacceptable.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://11149146]
Approved by LanX
Front-paged by kcott
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others musing on the Monastery: (7)
As of 2024-04-23 12:32 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found