http://qs321.pair.com?node_id=400708

johnnywang has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I'm looking for wisdom on the following searching/matching problem: I have two files, each containing some records (say one string per line). I'd like to go through the second file to find out whether each record is in the first file. One easy way to do it is to load the first as a hash, then just iterate through the second. My problem is the files are huge, putting the whole thing in memory is pushing the limits. Is there a less resource-demanding approach? of course speed is also a great concern. (The files are now containing about 10 million records, but growing. If the whole thing can be done within a few hours to a day would be fine.) Thanks.