Not wishing to rain on your parade, but what happens if one of the duplicate-pairs you are trying to eliminate is split over two of the files you created?

I think the advice you were given the first time you brought this up was probably closer to the right track.

Best of luck with it anyway. Do you mean `tyranny'?

my %seen;
while (my $line = <$fh>) {
    chomp($line);
    print "$line\n" unless seen{$line};
    $seen{$line}++;
}
[download]

Actually it does not take 1 second, I give the refresh a one second pause bettween each page. I should have been more clear. I am definitely not writing a SPAM program. I'm not a fan of SPAMMERs. Thanks for the enlightenment. TIURIC

I have a bad habit of doing this (that is, dodging the question), but for you this is the time to use a relational (SQL) database and force the email address field to be UNIQUE, or else do a "SELECT UNIQUE * FROM" or something like that.

Your flatfile technique is bound to be slower. 1200sec/60(sec/min) = 20 minutes. A database could fly through this sort of thing, and then you would have a system that would be better to write future tools on top of.

Postgre and MySQL are both great free DB servers available for most platforms. I tend to prefer Postgres for historical reasons about MySQL (most reasons no longer valid), both are rapidly imporving.