http://qs321.pair.com?node_id=894191


in reply to temp hold logfiles in memory?

You mention that your program is 4-5 seconds per minute slower. Depending on the total number of minutes, that may not be a lot of time in the long view. Compare the amount of time you will save over the lifetime of the programs use to the amount of time it will take to find the true source of the slow down and optimize it away. Unless you are running this program over and over, need real-time responsiveness or you have a marketing reason to look fast and slick, you may want to ask whether optimization is even necessary.

If you still feel you need to optimize, you might be able to confirm your suspicions about the cause by running your script with the -d:DProf option. This will generate a file named tmon.out in the current directory. This file contains raw profiling data. You can analyzed the contents of the file, by typing the command dprofpp, which should be part of your Perl installation. This will give you an idea of how much time is taken by each subroutine. Thus your shell would look something like this:

$ perl -d:DProf myscript.pl $ dprofpp

If you are still convinced that buffering is the source of your problem, you may want to consider using the "sys" family of functions. These are low level functions that work directly with the file and will allow you to define your own buffer size and buffering strategy:

If you go this route, do not use the normal Perl file io (open, read, print, seek tell) on the same file handle or your file handle will give you confusing and not terribly helpful results.

Before you make any changes, you should restructure your program so that the old file buffering code is encapsulated in a subroutine. The new code should also be in a subroutine. That way you can swap out the old and new code at will to compare them. Encapsulating the two approaches (Perl buffering vs. custom buffering) will also let you use a wonderful optimization tool, a core module built into Perl: Benchmark. This module lets you compare two subroutines to see which is faster. If you are going to put in the effort to optimize, it pays to make sure that you are actually making an improvement.

Finally, you may want to take a look at this Unix Review article, which goes over some of the basic tools for optimizing programs: Speeding up your Perl programs.

Replies are listed 'Best First'.
Re^2: temp hold logfiles in memory?
by deibyz (Hermit) on Mar 21, 2011 at 09:19 UTC

    Also, don't forget to have a look at Devel::NYTProf, which generates nice http reports :). Has saved me from a lot of hours of debugging and profiling...

Re^2: temp hold logfiles in memory?
by TRoderic (Novice) on Mar 20, 2011 at 12:32 UTC
    thank you *so much* for that, it's exactly the sort of information I was looking for.

    the program is internal, but likely to be repackaged and re-purposed a good many times in the near future, hence my attempts to get it as close to right now, before some new purpose highlights an old oversight.

    with the information from that subroutine timer, I would not be surprised if the actual process can be improved in the right directions!

    thanks again.