First: if you're using a distro supplied perl, make sure you've got the perl-debug package installed. Same goes for the debug package for libgcrypt.
Second: you can use valgrind to get at least some kind of indication of where any segfaults and badly allocated memory segments are located, if you've got that kind of problem.
Third: reduce the 00-test.t file to the absolute minimum where it'll still shows the problem. .xs files normally don't contain that much code and it should be possible to figure out where the problem is from any stack traces or manually tracing what calls are made across the perl/C boundaries. You can also put croak() or printf() calls in the XS code if you want more direct reporting of what's going on.