Several ways I can think of:

• See the CPAN testers "Author's FAQ"; there are some suggestions for how to deal with failures that result from missing libraries, or how to deal with OS platforms that you know are incompatible. If memory serves, the suggestion often is to have Makefile.PL detect incompatible environments and 'die', which will get you an NA instead of a FAIL.
• If it's just one or two testers, get in touch with them and see if they can work out what it is about their particular setup with is failing. And if you really have exhausted all alternatives, ask them to add your module to their ignore list (that is possible for them.... usually they'll do it without you even requesting it if your module install starts consuming astronomical amounts of resources, or starts crashing their smoker). I suspect that if you ask nicely, the one or two who are a problem for you would be willing to block-list your distribution on their smoker.

• Look at the individual smoke test reports for the failures you would like to eliminate. Is there something in common with those failures that isn't present in the ones that pass? Perhaps there's an environment variable (which will show up in the reports) that your Makefile.PL could detect, and if present, could die. If your Makefile.PL aborts, you'll get an NA from the smoke tester instead of a FAIL. You might use Config, and match against some known criteria that shows up in the failed smoke test reports. If that configuration is detected, 'die' within Makefile.PL.