Is it safe to assume you're really using system perl + a significant investment in modules installed as a non-privleged user?
Yes, I'm sorry if that wasn't clear enough in the description: XS modules are installed locally against system perl.
Are the XS modules darkpan (not on CPAN) or are they just the result of some heavy "tweaking"?
No they're all public XS modules. No tweaking.
It sounds like you want to recompile the XS modules "in place", maybe?
That would be the goal yes. Ideally I would "list" XS only distribution names in ~/local/perl/lib and then proceed again with the whole make/test/install cycle.