almr has asked for the wisdom of the Perl Monks concerning the following question:
I've been working on a "quick and dirty" script that should run from .bashrc. It seemed a lot slower than an equivalent bash script (which runs several executables...), so I decided to time it. It turned out the code itself didn't really make a difference -- it was the imports that were sluggish.
In particular, just importing (without using at all) autodie, Pod::Usage and Getopt::Long will slow down a do-nothing program. Here are the results from hyperfine (-adie means enable autodie, -podu = Pod::Usage, and -go = Getopt::Long; actual programs at the end)
Benchmark 1: /tmp/x-go-adie-podu.pl
Time (mean ± s): 107.1 ms ± 0.5 ms [User: 94.5 ms, Sys: 12.5 ms]
Range (min … max): 106.2 ms … 108.1 ms 27 runs
Benchmark 2: /tmp/x-go-adie.pl
Time (mean ± s): 50.2 ms ± 0.2 ms [User: 44.5 ms, Sys: 5.7 ms]
Range (min … max): 49.7 ms … 50.6 ms 58 runs
Benchmark 3: /tmp/x-go.pl
Time (mean ± s): 25.7 ms ± 0.1 ms [User: 21.6 ms, Sys: 4.1 ms]
Range (min … max): 25.5 ms … 25.9 ms 112 runs
Benchmark 4: /tmp/x.pl
Time (mean ± s): 7.8 ms ± 0.1 ms [User: 5.3 ms, Sys: 2.5 ms]
Range (min … max): 7.6 ms … 8.1 ms 324 runs
So, we jump from 8ms to 25ms to 50ms to 107ms just by importing (in order) Getopt::Long, autodie, and Pod::Usage. Once it reaches tenths-of-a-second territory I get itchy, because it's starting to become visible to humans.
Anyway
- why are these so slow to import?
- can they be easily replaced? I could ditch autodie; I could use perldoc to pre-generate the usage message; but I'd rather not hand-code an arg parsing loop.
Actual files:
### /tmp/x-go-adie-podu.pl
#!/usr/bin/env perl
use strict; use warnings;
use Getopt::Long qw( :config );
use Pod::Usage;
use autodie;
### /tmp/x-go-adie.pl
#!/usr/bin/env perl
use strict; use warnings;
use Getopt::Long qw( :config );
use autodie;
### /tmp/x-go.pl
#!/usr/bin/env perl
use strict; use warnings;
use Getopt::Long qw( :config );
### /tmp/x.pl
#!/usr/bin/env perl
use strict; use warnings;
Re: slow startup for some common modules? (autodie, Pod::Usage, Getopt::Long))
by hippo (Bishop) on Dec 28, 2022 at 10:15 UTC
|
why are these so slow to import?
Well, if you think about it when you use Pod::Usage it has to parse the source of your code, extract the POD and process the POD to present usage info - that will all take time. You can save this by postponing the load until/unless it is actually needed (most runs will never need it, of course). The other option is to use pod2usage out-of-band in advance and just copy the output of that into your code instead statically.
I find that Getopt::Std is much faster than Getopt::Long. Its reduced functionality may well be worth the trade-off for you.
| [reply] |
|
By "postponing until needed", do you mean something like eval q{use Pod::Usage}, whenever the help message is needed ?
Thanks for the pointer to Getopt::Std, I'll consider it the next time around. I've ended up coding a simple arg-parsing loop in the meanwhile (the repo is ssh_agent_share; quite trivial, but it seems I end up learning something new every time I reach back to perl).
| [reply] [d/l] [select] |
|
use Pod::Usage;
# is the same as
BEGIN {
require Pod::Usage;
Pod::Usage->import();
}
BEGIN blocks execute at compile time, not run time.
Assuming that often you run without ever using the Pod functions,
and you want to fire that module up only if you will be needing it,
you could have a runtime flag like this:
if ($needPod)
{
require Pod::Usage;
Pod::Usage->import();
}
I leave it you to decide how this applies to your code. You have to do the require and import before using any
functionality of Pod::Usage.
Usually GetOPt::Std is enough for me. I almost never use autodie and absolutely never use it in end user code.
| [reply] [d/l] [select] |
|
|
|
I use this recipe in most scripts:
use Getopt::Long;
sub pod2usage { require Pod::Usage; goto \&Pod::Usage::pod2usage; }
GetOptions(
...
...
) or pod2usage(2);
I don't mind paying for Getopt::Long because I like its features, but there's no reason to load Pod::Usage unless the user screws up the commandline.
On that note, someone ought to contribute a patch to Pod::Usage that avoids loading anything until it gets called. Then I could skip that bit of ugly boilerplate in my code. | [reply] [d/l] |
|
require Pod::Usage;
Pod::Usage::pod2usage( ... );
| [reply] [d/l] |
Re: slow startup for some common modules? (autodie, Pod::Usage, Getopt::Long))
by LanX (Saint) on Dec 27, 2022 at 22:22 UTC
|
Perl startup is "slow" by some measures.
It starts the engine, does a lot of initializing, has to load various modules from filesystem and if they are lots of pure Perl need to compile them, and probably need to repeat all this to import more dependencies.
So many things to consider, such that your benchmarks should be more detailed, probably using the NYT profiler.
I don't think anything below 0.3 secs is really worth worrying, but if you really need to run a program many times you should consider talking to a server with everything pre-initialized. That's the trick behind fast-cgi etc... You'll trade space for time like that.
But if that's not an option because you have too many different programs which need boosting, you could consider bundling all dependency into one big package with par-packer and precompiling them into a .plc with bytecode and store them in a ram-disk.
Disclaimer: Never tried that myself, cause as I said, Perl startup is usually fast enough for me. (perltidy probably being the only exception)
| [reply] |
|
Well, as I've noticed many times before, and as the hyperfine above shows, the actual startup cost of perl itself is not bad at all (a few msec). For me it's only these three modules that really bite (in particular Pod::Usage, unused, takes 50msec). I had no problem with, for example, Time::HiRes, Fcntl, or Sys::Hostname.
In my case, I think it makes sense to replace the slow modules (though I'm still surprised they are that slow). I've tried pp to create an executable (which seems to be a self-extracting zip) and it ran about three times slower.
| [reply] [d/l] [select] |
Re: slow startup for some common modules? (autodie, Pod::Usage, Getopt::Long))
by kcott (Archbishop) on Dec 29, 2022 at 02:55 UTC
|
G'day almr,
I made verbatim copies of your four scripts and ran them using time (I don't have hyperfine).
I performed multiple runs on each; I got results that were of the same order of magnitude as yours.
So, there's nothing problematic with your system in this regard.
I see some workarounds have been suggested.
I use the three modules you tested in most of my $work scripts; I've never had any comments about slowness.
I don't imagine any of your users will complain that a script took 100ms to start running. :-)
You'll no doubt be loading additional modules which will increase the startup time.
Keep an eye on what you're importing.
# Import nothing -- fastest but requires additional coding
use Some::Module ();
...
Some::Module::some_function();
Some::Module::other_function();
# Import just the parts you intend to use -- still fast; less coding
use Some::Module qw{some_function other_function};
...
some_function();
other_function();
# Import everything -- could be very slow
use Some::Module;
...
any_function();
If you're really concerned about startup speeds, put something like this at the start of your script:
BEGIN { print "Loading app (could take a second or two) ...\n"; }
Then, when it only takes 100ms or so, your users should be very pleased.
| [reply] [d/l] [select] |
|
Thanks for taking a look. For a standalone "application", 100msec for sure doesn't matter. However, if (1) I'm calling several perl scripts inside bashrc, or (2) a script is invoked after every command (PROMPT_COMMAND in bash) or (3) a script runs in an editor on every keystroke (see the vim-like editor Kakoune), those 100msec add up. 10 scripts taking 100msec each result in a 1sec delay before being able to type anything, which is of course unacceptable.
| [reply] |
|
|