Thanks guys for the insightful input, I agree a more robust and elegant end solution would be to have SWIG auto-generate the appropriate Makefile.PL (or Build.PL) that will then be used to do the compilation and linking properly for one's environment.
I will work on this and present it to the SWIG people. For the time being they have at least updated the Perl quick tutorial which was broken to what I proposed and this has been tested. I know its only for systems with gcc but that is what their original commands had anyway so I guess no harm done.