saintmike has asked for the wisdom of the Perl Monks concerning the following question:

I really like the CPAN smoke test efforts, automatically sending me feedback for platforms I'm not testing my releases on.

Only problem is that some things are meant to be broken, and I don't want to get email to tell me that they are. For example, some modules just don't work on Windows because of OS limitations. What am I as the CPAN author supposed to do about it?

If I mark the tests as "SKIPPED" on these platforms, the smoke tests will show up as PASS and module consumers might be tricked into thinking that it's OK to use the module on the broken platform.

On the other hand, if I don't do anything about it, I look somewhat foolish with my broken tests and keep getting emails from the smokers.

Is there a way to say "This is not supposed to be working, don't use it on this platform"?

  • Comment on CPAN Smoke Tests: Some platforms are meant to be broken

Replies are listed 'Best First'.
Re: CPAN Smoke Tests: Some platforms are meant to be broken
by Corion (Patriarch) on Aug 21, 2011 at 20:30 UTC

    I think if your tests are inapplicable, the best approach is to bail out from Makefile.PL with an exit code of 1 and the message "BAIL OUT!" (or something like that) on STDERR. Maybe it's also enough to just not create a Makefile. Have a look around at other modules that are OS-specific. For Win32::Wlan, I use the following boilerplate code in the tests skip all the tests unless they are applicable:

    #perl -w use strict; use Test::More; BEGIN { if ($^O !~ /Win32/i) { plan skip_all => "Win32::Wlan only works on Win32"; } else { plan tests => 1; } }; ...
      ... but according to the smoke test reports, there's green light for GNU/Linux, OpenBSD, NetBSD, and Solaris :).

      Wouldn't you rather have a clear indication that it doesn't work on these platforms?

        I think there is some way of telling CPAN testers to emit n/a for those operating systems, but if somebody installs a module with Win32:: in its name on a system that is not Win32 (or Win64), I don't think that passing tests are their main problem :)

Re: CPAN Smoke Tests: Some platforms are meant to be broken
by DrHyde (Prior) on Aug 22, 2011 at 13:04 UTC

    The tool you are looking for is Devel::AssertOS. If you use that in Makefile.PL, then CPAN testers will magically bail out if they're running the wrong platform. The use-devel-assertos script provides a convenient method for updating your Makefile.PL and bundling all the relevant modules.

    Within your code, if you want to do different things depending on the platform - for example, you need to write completely different code on Windows and Unix for figure out who the current user is - then Devel::CheckOS is what you want.

Re: CPAN Smoke Tests: Some platforms are meant to be broken
by eyepopslikeamosquito (Bishop) on Aug 21, 2011 at 21:22 UTC

    I suppose it would be possible to extend the CPAN META.json/META.yml file. Maybe something like "supported_platforms" and/or "unsupported_platforms". Precisely how you define a "platform" seems an awkward problem though, e.g. "Windows" versus "Windows Vista"; "Unix" versus "Linux" versus "AIX 6.1" versus "AIX 5.3" and so on. Also, Perl may be ported to new platforms after a module is released.

      Right, and a module might just magically work on a platform that the author never intended it to function on. Or a new platform ("QuantComp128"), might spring into existence which it might work on, although the author never heard of it.

      I like your idea of using "unsupported_platforms" meta info that lists platforms that the author knows the tests will fail on, but considers that normal or doesn't care.

      unsupported_platforms: - Win32
      would then alert the smokers to mark the appropriate column in the matrix specially and refrain from sending out emails to the authors with the failed test suite results, because the results are actually considered to be a fulfilled prophecy.

        I'd prefer the use of code to determine if the platform is unsupported over trying to standardize a data format for such declarations [for example, you might declare a platform as unsupported because it lacks the quantumflux.h file or because the quantumflux.h files does not define entangle() or because there is no /proc/torhock present].

        Plus, the use of code to make such determinations is already supported, it just isn't documented sufficiently conveniently.

        - tye        

Re: CPAN Smoke Tests: Some platforms are meant to be broken
by szabgab (Priest) on Aug 23, 2011 at 19:09 UTC
    If your modules cannot work on certain OS-es then you can indicated that in the Makefile.PL/Build.PL

    If certain tests don't work then you can skip them.

    See the CPAN Author Notes on the CPAN Testers wiki for details.