I think you're probably noticing the cost of global destruction. At the very end of the program's execution, the interpreter will go through every single allocation and see if it needs to be DESTROYed. See perlobj, especially the bit at the end.

I don't think you can switch this off. If anything, you can only make it worse :) A perl binary built with -DDEBUGGING will let you set the PERL_DESTRUCT_LEVEL to increasing values, which will exercise the garbage collector more and more severely. This is used to smoke out problems involving memory leaks.

One avenue worth exploring is whether in fact you are leaking a colossal amount of memory during the program's execution, thus leaving 5.4Gb to be cleaned up at the end. Remove the leak, and maybe global destruction will become much cheaper.