Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options

Profiling forking code?

by Jeppe (Monk)
on Jan 13, 2003 at 18:20 UTC ( #226544=perlquestion: print w/replies, xml ) Need Help??

Jeppe has asked for the wisdom of the Perl Monks concerning the following question:

Fellow monks, I have a question for you. How do I best profile a forking program? It is a daemon, and all the heavy computation is performed in the child processes that are spawned. I also do some slightly messy things with string evals that load classes, and then more string evals to get an object from that class.

My question is essentially "How do I figure out what to speed up?". I think profiling is a good start since the code is almost mature and I would rather not redesign unless I can document that I need to.

Replies are listed 'Best First'.
Re: Profiling forking code?
by dws (Chancellor) on Jan 13, 2003 at 19:31 UTC
    How do I best profile a forking program? It is a daemon, and all the heavy computation is performed in the child processes that are spawned.

    Do you have enough of a test harness in place that you can invoke the child portion directly, without forking? I would think that would make profiling far, far simpler.

      Hmm. It's never that easy.

      As it turns out, DProf segfaults while Devel::Profiler screws up the file when confronted with my scary class definition logic. What I'm doing is VERY insecure, so only do this if you TRUST your users.

      I've designed it so that one general class is a shell around the class I load. I've done this for several reasons

      1. If there is a syntax error in one of the classes, I don't want bring down the entire process.
      2. I want to give my customers the chance to create their own code that ties in, without altering the core code
      3. I got a performance improvement when using this method over regular inheritance
      Anyhow, I have a class. The objects have an associated file, and that file contains part of the code - using a standard API. So, I create a header and a footer, and insert the code from the file inbetween. Then, I run a string eval around that, and then I have a new child class. I create an object of the child class, and keep it as a variable in the parent class. Inside the parent class, I call that object where appropriate.

      As it turns out, the profilers do not approve of this.

      Any brilliant ideas, fellow monks? Need I rewrite, or do you see a way for me to work around my problem?

        According to perldebguts:
        A hash %DB::sub is maintained, whose keys are subrou­tine names and whose values have the form "file­name:startline-endline". "filename" has the form "(eval 34)" for subroutines defined inside "eval"s, or "(re_eval 19)" for those within regex code assertions.

        It seems to me your profiler would be happier (and you would find the output more usable) if you pre-created the object packages and used them from files instead of creating them inside eval.

      I love it when other people are as right as you. It's probably just a matter of copy and paste, or maybe even sticking the functions inside the daemon script into and use that.

      I'll update you on the results.

Re: Profiling forking code?
by Anonymous Monk on Jan 13, 2003 at 19:01 UTC
    I have not tried this yet, but Devel::AutoProfile looks promising.

    From the documentation:
    ... You can supply a callback which will be called at END time. This code reference will be passed a hash containing all the caller information in the form of a hash. Your callback routine can the do whatever it wishes with this information.

    So, my suggestion would be to fix up the callback so that it writes to different places for each forked child.

      Cool looking module!

      The only problem I see for this application is how to avoid "double-counting" (or worse) things that happen before the fork(s).

      But the guts of D::AP seem remarkably straightforward, so adapting it to solve this problem shouldn't be impossible.

      That module looks promising, and it appears to be a pure perl module which alleviates several problems for those using AS. Unfortunately it requires Timer::HiRes::ualarm(), which isn't implemented on the Win32 platform. Drat!

      Examine what is said, not who speaks.

      The 7th Rule of perl club is -- pearl clubs are easily damaged. Use a diamond club instead.

Re: Profiling forking code?
by BrowserUk (Patriarch) on Jan 13, 2003 at 18:45 UTC

    The same question came up a few months ago at Profiling a forking program, but I just took a quick scan of that thread and it didn't seem to suggest any good resolution.

    The biggest problem seems to be that DProf uses a hard-wired filename for the tracing file 'tmon.out', When I took a look at doing the same thing a little while ago, I thought that I might be able to modify the module so that it incorporated the process ID (or similar) into the output filename, but when I tried to track down the location of the filename, I found it embedded in a /lib/auto/DProf/Dprof.dll which stopped me persuing the idea further.

    If you built your own version of perl, you might have more luck in modifying the C source of the .dll (or your binary equivalent. I did look at it, but it make heavy use of macros and was non-obvious (to me) where to start.

    Maybe one of the other Profiling modules I seen mentioned recently but don't recall the name of right now is or could be made more process freindly.

    Good luck.

    Examine what is said, not who speaks.

    The 7th Rule of perl club is -- pearl clubs are easily damaged. Use a diamond club instead.

      $ENV{PERL_DPROF_OUT_FILE_NAME} = "foo${$}.out"; is how to change that. I've never tried profiling over a fork, though.

      It looks as if the child will have to be started with that environment variable localized.

      Update: I have tried it now, couldn't work it without going through the shell,

      #!/usr/bin/perl -d:DProf { local $ENV{PERL_DPROF_OUT_FILE_NAME} = "foo.out"; my $cpid = open my $foo, '|-', '/usr/bin/perl -d:DProf' o +r die $!; } print 'In Parent', $/ or die $!; sleep 10;
      #!/usr/bin/perl print 'In Child',$/ or die $!; sleep 10;
      $ ./
      In Child
      In Parent
      $ ls *.out
      foo.out tmon.out

      After Compline,

        Yes, I discovered that to, but couldn't work out how you might cause fork to start a new copy of the debugger with that variable changed.

        I did think that as AS forks are implemented as threads of the same process, by modifying module so that it incorporated the threadID in the name of the output file at runtime, it might be possible to get a new output file for each process (thread), but as I said, when I loked at the C-source for the .DLL, I found myself going in circles trying to unwind the macros involved and gave up. Even ignoring the macros problem, it was not at all obvious to me whether the debugger could be coerced into using a different handle within each thread, nor whether it would be threadsafe. I was looking at the source for AS 5.6.1 at the time.

        In the end, I started playing with the idea of using Benchmark::Timer dynamically required by each child process after it was forked and outputting the results to files opened by the child processes after forking incorporating the psuedo-process ID, but I found my problem before persuing that idea any further. The biggest problem with that idea is that you end up having to modify the program under test in order to profile, which is less than desirable.

        Examine what is said, not who speaks.

        The 7th Rule of perl club is -- pearl clubs are easily damaged. Use a diamond club instead.

Re: Profiling forking code?
by RMGir (Prior) on Jan 13, 2003 at 19:27 UTC
    I'm wondering if you could do some AUTOLOAD magic on a few modules.

    You could replace the "real" method/sub names with some prefixed form, and then use AUTOLOAD to generate stubs that log what's being called with high-res start time, call prefixed form, then log end time. You should probably have that AUTOLOAD stub log to per-pid logfiles, to avoid having to lock things.

    If your modules are "regular" enough, you could even write a perl script to instrument your modules :)

    This is pretty well what a profiler would do for you, but since you're forking, I think you might have to do it for yourself...

    Of course, this involves modifying the module sources. :( But if you're not profiling, you could have the AUTOLOAD sub alias *outerform=&innerform.

Re: Profiling forking code?
by jimc (Sexton) on Jan 13, 2003 at 20:36 UTC
    this is something of a punt, but nobody else has suggested it. Add an option to your program to operate in non-forking mode. This will allow you to:

    profile your code,
    run it in debugger (good to have, just in case ;-)

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://226544]
Approved by Helter
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (4)
As of 2022-01-24 12:39 GMT
Find Nodes?
    Voting Booth?
    In 2022, my preferred method to securely store passwords is:

    Results (64 votes). Check out past polls.