Ask yourself whether the efficiency benefits of running C/XS code are really required or not. Then decide if there's any functionality written in C/XS that can't run in Pure Perl without compromising the quality of the distribution.

It's always easier to deploy a distribution cross-platform if there's no XS for sure, but there are those times where the XS is definitely required.

If it's not required, and the benefits won't even be noticed, I leave it out.