I've long disliked controlling the running of Perl CPAN module tests via various ad hoc environment variables, such as AUTOMATED_TESTING and TEST_AUTHOR. Somewhat related is the make disttest action, the xt/ directory for "extra" (typically author) tests, and the t/test_manifest file (see Test::Manifest).

I'm a bit rusty on all this stuff, so I did a bit of googling (see References section below) but was unable to find any definitive reference on these issues.

Just as a CPAN module has metadata (META.yml) how about generalizing things like TEST_AUTHOR and xt/ by inventing a standard set of more comprehensive metadata for CPAN tests? Apart from things like TEST_AUTHOR, there is all sorts of other metadata that could be used to describe tests, such as how long the test takes to run, what platforms are supported, and many more. For this to work well, I suspect you'd need a central authority to define the definitive (closed) set of test metadata names, plus a mechanism for growing these names over time.

A simple way to implement this would be by embedding the test metadata in the .t file itself, for example:

# file t/sometest.t
# Here is the metadata for test t/sometest.t
=begin perltestmeta

LONG_RUNNING=1
RELEASE_TEST=1
AUTHOR_TEST=1
SUPPORTED_PLATFORMS=Unix,Windows
UNSUPPORTED_PLATFORMS=VMS
# ... other test metadata as required

=end perltestmeta

=cut

use Test::More;

# Perl tests follow...
[download]

CPAN tools would need to be updated to use this metadata, of course.

I'm interested in getting early feedback on this idea before developing it further if appropriate (presumably by posting to perl-qa@perl.org mailing list).

References

• perl-qa mailing list post "externalizing test metadata" 19 Aug 2007 by Eric Wilhelm
• perl-qa mailing list post about author tests
• Alias use.perl journal: The Oslo Consensus
• Alias use.perl journal: Best Practice for Author Tests
• Schwern use.perl journal: CPAN Version Advisories
• CPAN Testers Wiki: Notes for CPAN Authors
• dagolden Perl talks (see "Hackers gone wild: Spring break in Oslo" for Oslo Consensus)
• CPAN::Testers::Metabase
• Debian mailing list post about author tests by Jonathan Yu
• make disttest action
• W3C Test Metadata - W3C Working Group Note 14 September 2005
• CPAN Smoke Tests: Some platforms are meant to be broken

• Test::Manifest
• Test::XT
• Module::Install
• Module::Install::ExtraTests
• Module::Install::AuthorTests
• Module::Faker
• Devel::CheckOS (includes Devel::AssertOS)
• Devel::CheckLib
• Probe::Perl

• The Lancaster Consensus
• The Annotated Lancaster Consensus

The Oslo Consensus (May 2008)

See The Oslo Consensus for full details.

• $ENV{AUTOMATED_TESTING} - not $ENV{PERL_AUTOMATED_TESTING}
• $ENV{RELEASE_TESTING} - not $ENV{AUTHOR_TESTING} or $ENV{PERL_AUTHOR_TESTING}
• xt/ directory for release and other non-install-time tests (subdirectories optional)
• Support 'requires => { perl => 5.xxx }' and extend to to all 'requires' types
• *.PL should generate META_LOCAL.yml with requirements after dynamic configuration

The Lancaster Consensus (April 2013)

See The Lancaster Consensus and The Annotated Lancaster Consensus and cpan.io for full details.

Historically, AUTOMATED_TESTING has been confusing, used for a number of different purposes:

1. I don't want the user to interact with this test.
2. This is a long-running test.
3. This test depends on an external website (say) and I don't want to stop the user installing if it fails, but I want to see what CPAN smokers experienced.

The Lancaster Consensus clarifies the semantics of AUTOMATED_TESTING and RELEASE_TESTING and adds three new environment variables, making a total of five:

• AUTOMATED_TESTING
• NONINTERACTIVE_TESTING
• EXTENDED_TESTING
• RELEASE_TESTING
• AUTHOR_TESTING

To run module tests after installation, use new target "make test-installed", equivalent to "make test" but without adding blib to @INC.

Update: There is also NO_NETWORK_TESTING which, if set, should prohibit anything relying on internet access (see here).

The Berlin Consensus (2015) and PTS Oslo (2018)

• The Berlin Consensus Recommended standards of care for CPAN authors
• notes from the review at PTS 2018 Oslo comments by Karen Etheridge et al

Module::Install::ExtraTests

The Module::Install::ExtraTests plugin adds one Module::Install command (extra_tests;) which declares that the test files found in the directory ./xt should be run only in certain instances:

• ./xt/author - run when the tests are being run in an author's working copy
• ./xt/smoke - run when the dist is being smoked (AUTOMATED_TESTING=1)
• ./xt/release - run during "make disttest"

These directories are recursively scanned for *.t files. If any directories or files exist in ./xt that are not recognized, the Makefile.PL will die.

References Added Later

• CPAN::Distroprefs - mentioned by Tux here (e.g. see ack-prefs and Andreas distroprefs). Search for distroprefs in CPAN module doco.
• Possible metadata: NEEDS_ROOT (see this reply by EvanCarroll)

• Let's try for a better CPAN experience by cavac (2021)
• Testing Environment Variables by hippo (2021) (clarifies AUTHOR_TESTING v RELEASE_TESTING ...)
• Module Change Log by Bod (2022) (lots of useful replies)
• How to author a module with optional XS by NERDVANA (2023)

• Test::Is
• Test::DescribeMe
• Test::Kwalitee
• CPAN::Meta::Spec
• Release::Checklist

• CPANTS testing service for CPAN distributions
• CPANTS Kwalitee Indicators

• Perl Toolchain Documents (CPAN.io by BooK and NeilB) (Oslo Consensus, Lancaster Consensus, Berlin Consensus, Lyon Compromise from Perl QA Conference)
• toolchain-site - (github) cpan-common, berlin consensus, lancaster consensus, lyon amendment, lyon compromise, oslo consensus

• Effective Automated Testing
• Properties, metadata and Perl celebrities (discusses adding properties aka metadata to A::E)
• Metadata (wikipedia)
• Resource Description Framework (wikipedia)

Updated 29-Aug: Added links for CPAN modules Test::XT, Module::Install::ExtraTests et al. Later: Added more references, including the Oslo Consensus, Lancaster Consensus, Berlin Consensus, PTS Oslo.

---

---

• Are you posting in the right place? Check out Where do I post X? to know for sure.
• Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:

  <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>

• Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
• Want more info? How to link or How to display code and escape characters are good places to start.