For just comparing files, it might be enough to do a CRC32 and use this along with the file size. This has the added advantage of allowing quick scans through ZIP files for dupes as well (using Archive::Zip of course) because ZIP files already have crc32 values in them.

The CRC32 found in Compress::Zlib runs 82% faster than Digest::MD5 on my system, using the following benchmark program:

#!/usr/bin/perl -w
use strict;
use IO::File;
use Compress::Zlib ();
use Digest::MD5;
use Benchmark;
use constant BUFSIZE => 32768;

sub crc32
{
 my $fh = shift;
 binmode($fh);
 sysseek($fh, 0, 0); # rewind
 my $buffer = ' ' x BUFSIZE;
 my $crc = 0;
 while ($fh->sysread($buffer, BUFSIZE))
 {
  $crc = Compress::Zlib::crc32($buffer, $crc);
 }
 return $crc;
}

sub md5
{
 my $fh = shift;
 seek($fh, 0, 0); # rewind
 my $md5 = Digest::MD5->new();
 $md5->addfile($fh);
 return $md5->digest;
}

foreach my $file (@ARGV)
{
 my $fh = IO::File->new($file);
 binmode($fh);
 next if !defined($fh);
 Benchmark::cmpthese(-10, {
  "crc32 $file", sub { crc32($fh) },
  "md5 $file", sub { md5($fh) }
 });
}
[download]