There are a number of sources of information on why this has crept into the preamble of many modules, but the authoritative source is probably perlmodstyle, where it says this:

A correct CPAN version number is a floating point number with at least 2 digits after the decimal. You can test whether it conforms to CPAN by using

    perl -MExtUtils::MakeMaker -le 'print MM->parse_version(shift)' \

                                                            'Foo.pm'
[download]

If you want to release a 'beta' or 'alpha' version of a module but don't want CPAN.pm to list it as most recent use an '_' after the regular version number followed by at least 2 digits, eg. 1.20_01. If you do this, the following idiom is recommended:

  our $VERSION = "1.12_01"; # so CPAN distribution will have
                            # right filename
  our $XS_VERSION = $VERSION; # only needed if you have XS code
  $VERSION = eval $VERSION; # so "use Module 0.002" won't warn on
                            # underscore
[download]

With that trick MakeMaker will only read the first line and thus read the underscore, while the perl interpreter will evaluate the $VERSION and convert the string into a number. Later operations that treat $VERSION as a number will then be able to do so without provoking a warning about $VERSION not being a number.

Never release anything (even a one-word documentation patch) without incrementing the number. Even a one-word documentation patch should result in a change in version at the sub-minor level.

So it's a trick to allow ExtUtils::MakeMaker to be able to deal with underscores in development-level release version numbers, while still letting numeric checks of version numbers work. The reason it works is because Perl allows underscores in numeric literals, but to get that treatment, the numeric literal needs to be evaluated by the interpreter. We need to preserve the underscore in the string for EU::MM (so can't make it a literal at compiletime), but we need it to get that special treatment eventually. That requirement is solved by doing the string eval.

For a subset of possible numeric representations a substitution regular expression such as this would be equally adequate: $VERSION =~ tr/_//d;. But I imagine that is inadequate for some of the edge cases of what Perl is able to convert to a number. There's a purity to shrugging and saying "Rather than trying to solve it through string manipulation, let Perl solve it the way Perl already knows how to."

This is another unfortunate bit of "tribal knowledge" discussed in one of SawyerX's TPCiC talks. Wouldn't it be great if EU::MM could look elsewhere to decide if this is a development release, or if use Foo 0.02 already dismissed underscores from $VERSION. But then this might break some esoteric machinations performed by modules already on CPAN, so in an effort to avoid breakage we inflict confusion through another piece of knowledge that raises the barrier of entry a little higher.

Update: There are, of course, other authoritative sources of information on how the versioning works, and these can provide additional context as to why the eval $VERSION code is common:

• ExtUtils::MakeMaker: The part that documents VERSION_FROM
• CPAN::Meta::Spec - Version Formats

These don't explicitly mention the underscore significance, but do discuss how EU::MM finds version numbers, and what constitute legal version numbers.

Additionally, there's the oracle itself: The venerable CPAN documentation: The Perl programming Authors Upload Server (PAUSE): Developer Releases. This oft-missed document needs to be required reading for CPAN contributors.

What is not immediately clear when reading that section of the document is that make dist incorporates the version number into the distribution tarball's filename. So if there's an underscore in the version number, there will be the requisite underscore in the filename, hinting the CPAN / PAUSE indexer that this is a development release, not to show up in the mainline index. It also informs that there are "other ways to do it." Putting -TRIAL in the distribution tarball's filename immediately before the file extension would also cause it to be marked as a development release. This can be accomplished in the Makefile.PL by including a DISTVNAME field with the version, including that -TRIAL bit, but this doesn't seem to be a very common approach. It could also be gamed with the DISTNAME field, but then the author has to be really careful to meet proper semantic requirements.