I love segfaults - anything that dumps core gives you a major headstart for debugging.

Before you start, think about what the program was asked to do: try to be clear in your mind what inputs it was given, and what it should have been trying to do. If you have information on screen grab it now before it scrolls off.

If the inputs are sufficiently constrained that you feel any problems should be reproducible, try reproducing it: do it now, while the steps are fresh in your mind.

Have a perl built with debugging symbols; I usually build with:

  ./Configure -des -Doptimize='-g -O2'
[download]

.. but setting just '-g' will make the code a bit easier to step through. Keep the build tree handy, so that gdb can find the source files. If you can afford to, use this as the default perl - if you can't reproduce the problem it can be frustrating having a core file only for a non-debug build, but the debug build will be bigger and slower¹.

Point gdb at the core file, examine the stack trace to get an idea of where we've come from, then look at the current source line and examine the variables it's accessing: the direct cause is usually obvious, but this line is rarely where the bug is. (In my experience, the actual bug is mostly one or two steps of indirection away: "over there we allow the pointer to become garbage; when we get here we dereference it" is commonest.)

Assuming we're not looking at the line with the bug on it, if the crash is not reproducible it now gets difficult: consider instrumenting the code to show before/after values anywhere the bad data might get modified, in the hope that next time the problem occurs it will reveal more information about the cause. Or just examine such areas of code, and hope to understand enough to be able to spot where there is an incorrect assumption or bad logic.

If it is a reproducible crash, you can save time now by cutting the test case down. You're not trying to create a minimal test case at this point (I've wasted many hours forgetting that), just trying to make it easy to get to the point where the crash occurs (or some relevant point before that) - try to hardcode data that was coming from external sources (and in particular anything from the keyboard), and try to cut out irrelevant code that takes more than a few seconds.

Then use breakpoints, and inspect the code and data, to zero in on the cause of the problem. I can't give much specific advice here - the path you take will depend entirely on what you discover, and you'll need to have (or develop) a good understanding of C, and the way C is used in the perl interpreter, and possibly XS if you're trying to debug modules using it.

One trick I have found useful is to set a breakpoint on a problem function, set it to ignore many times, then run the test case and see where it stops:

(gdb) break S_regmatch
Breakpoint 1 at 0x810d0be: file regexec.c, line 2272.
(gdb) ignore 1 100000
Will ignore next 100000 crossings of breakpoint 1.
(gdb) run
[...]
(gdb) info break
Num Type           Disp Enb Address    What
1   breakpoint     keep y   0x0810d0be in S_regmatch at regexec.c:2272
        breakpoint already hit 1713 times
        ignore next 98287 hits
[download]

Now you know how many times the break point was reached, you can set the ignore count to 1 or 2 less than that to stop it at a useful point.

Hope some of this is helpful,

Hugo

¹: C code doesn't run slower just because you have debugging symbols around, but Configure will spot -g in the C flags, and turn on -DDEBUGGING which has a fair amount of overhead. I don't know how to stop it doing that.