Debugging forking perl script? GDB?

cavac has asked for the wisdom of the Perl Monks concerning the following question:

I'm working (as most of the time) with my preforking webserver. I'm running into an issue where sometimes (after minutes/hours/days) one of those preforked processes stops responding and hovering at 100% CPU load. Perl 5.30.1 in Ubuntu 18.04.

All i need is a nice Perl stacktrace so i can tell whats going on. Starting the debugger on the command line isn't really an option, because the process forks hundreds of times. I can't run anything that world fork a gazillion terminals because the problem only happens on a server, and doing potentially hundreds of xterms via X11 forwarding is just not going to finish before the heat death of the universe.

I've also tried App::Stacktrace and Enbugger, both of which don't work. Since i suspects that the bug is somewhere in my exception/error handling code, i'd rather not introduce extra exceptions or signal handling code anyway, but i want to stop my process and dump the stacktrace.

Does anyone know how to dump the stacktrace of the script (not the interpreter) with gdb?

Are there any alternative solutions?

Update: I managed to find the problem by implementing a poor mans stacktrace. I wrote a pair of scripts to add/remove a debug print at the beginning of every function. After i roughly found where the code went bad, i spent a couple of days doing a static analysis. As suspected, the problem was in the error handling code for a block eval. You can find the scripts here and here.

perl -e 'use MIME::Base64; print decode_base64("4pmsIE5ldmVyIGdvbm5hIGdpdmUgeW91IHVwCiAgTmV2ZXIgZ29ubmEgbGV0IHlvdSBkb3duLi4uIOKZqwo=");'

Comment on Debugging forking perl script? GDB? Download Code

Replies are listed 'Best First'.
Re: Debugging forking perl script? GDB? by Corion (Patriarch) on Nov 18, 2019 at 13:11 UTC
I haven't used it, but App::Stacktrace claims to do just that. Looking at the source, it basically seems to automate gdb to step through things, and it seems to only know about Perl up to 5.14, so some manuall massaging of the code might be needed. There is Debugging Perl scripts which use fork(), and there, Debug::Fork::Tmux is mentioned, which doesn't use Xterms on the client but Tmux sessions on the server.	[reply]
Re^2: Debugging forking perl script? GDB? by cavac (Parson) on Nov 18, 2019 at 13:43 UTC
Yeah, i couldn't get App::Stacktrace to work. And i know exactly nothing about the internals of the perl interpreter, so i wouldn't even know how to fix the module. `perl -e 'use MIME::Base64; print decode_base64("4pmsIE5ldmVyIGdvbm5hIGdpdmUgeW91IHVwCiAgTmV2ZXIgZ29ubmEgbGV0IHlvdSBkb3duLi4uIOKZqwo=");'`	[reply] [d/l]
Re: Debugging forking perl script? GDB? by martin (Friar) on Nov 19, 2019 at 09:18 UTC
You stated you don't want to introduce signal handling code but a signal handler could actually be the least intrusive approach. Once per process, like in the code that runs in each freshly forked child, run something like: `use Carp (); $SIG{'QUIT'} = \&Carp::confess;` [download] It would be unusual for other code employing signals to use this particular signal for other purposes. Then, next time you find a looping process, kill it with SIGQUIT. A bash commandline to kill the process with PID 12345 could be: `kill -QUIT 12345` [download] The misbehaving instance of your program will die after dumping the stack trace on STDERR.	[reply] [d/l] [select]
Re: Debugging forking perl script? GDB? by xiaoyafeng (Deacon) on Nov 19, 2019 at 02:59 UTC
I took a look App:Stacktrace, It's compiled gdb to perl interpreter. It should work for you unless the internal of perl has been changed much. But your issue is different, theretcially, when fork create a new process, gdb will only monitor parent process, but your issue happen in child. So I suggest you writing a monitor script to solve this: when your program fork a new process, just sleep a second, find pid of new process, then use App::Stacktrace/gdb attach it. I am trying to improve my English skills, if you see a mistake please feel free to reply or /msg me a correction	[reply]
Re: Debugging forking perl script? GDB? by Anonymous Monk on Nov 18, 2019 at 13:04 UTC
So is it a fork bomb or is it a fork bomb? Is it swapping?	[reply]
Re^2: Debugging forking perl script? GDB? by cavac (Parson) on Nov 18, 2019 at 13:40 UTC
No fork bomb. Just sometimes one of the children decides to loop forever somewhere... `perl -e 'use MIME::Base64; print decode_base64("4pmsIE5ldmVyIGdvbm5hIGdpdmUgeW91IHVwCiAgTmV2ZXIgZ29ubmEgbGV0IHlvdSBkb3duLi4uIOKZqwo=");'`	[reply] [d/l]
Re^3: Debugging forking perl script? GDB? by Anonymous Monk on Nov 21, 2019 at 08:36 UTC
Include the PID and workload information (request url, rpc method name, queue item details, etc) in your logs along with start/end timestamps for processing the work. You can then look up the PID in the logs and find out what that pesky pegged worker is up to.	[reply]
Re: Debugging forking perl script? GDB? by Anonymous Monk on Nov 20, 2019 at 23:49 UTC
Crazy idea here: if you know that requests should complete in less than so-much time, could you somehow use a timer interrupt to cause the process to abort and dump itself?	[reply]
Re^2: Debugging forking perl script? GDB? by cavac (Parson) on Nov 21, 2019 at 09:56 UTC
That would have been an option before websockets. These days a web connection could last a really long time. `perl -e 'use MIME::Base64; print decode_base64("4pmsIE5ldmVyIGdvbm5hIGdpdmUgeW91IHVwCiAgTmV2ZXIgZ29ubmEgbGV0IHlvdSBkb3duLi4uIOKZqwo=");'`	[reply] [d/l]

Back to Seekers of Perl Wisdom