comment on

For just comparing files, it might be enough to do a CRC32 and use this along with the file size. This has the added advantage of allowing quick scans through ZIP files for dupes as well (using Archive::Zip of course) because ZIP files already have crc32 values in them.

The CRC32 found in Compress::Zlib runs 82% faster than Digest::MD5 on my system, using the following benchmark program:

#!/usr/bin/perl -w
use strict;
use IO::File;
use Compress::Zlib ();
use Digest::MD5;
use Benchmark;
use constant BUFSIZE => 32768;

sub crc32
{
 my $fh = shift;
 binmode($fh);
 sysseek($fh, 0, 0); # rewind
 my $buffer = ' ' x BUFSIZE;
 my $crc = 0;
 while ($fh->sysread($buffer, BUFSIZE))
 {
  $crc = Compress::Zlib::crc32($buffer, $crc);
 }
 return $crc;
}

sub md5
{
 my $fh = shift;
 seek($fh, 0, 0); # rewind
 my $md5 = Digest::MD5->new();
 $md5->addfile($fh);
 return $md5->digest;
}

foreach my $file (@ARGV)
{
 my $fh = IO::File->new($file);
 binmode($fh);
 next if !defined($fh);
 Benchmark::cmpthese(-10, {
  "crc32 $file", sub { crc32($fh) },
  "md5 $file", sub { md5($fh) }
 });
}
[download]

In reply to Re: Re: Find duplicate files. by bikeNomad
in thread Find duplicate files. by salvadors

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


laziness, impatience, and hubris
	PerlMonks