Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

Text difference

by Eye beauty (Initiate)
on Nov 21, 2007 at 12:22 UTC ( [id://652114]=perlquestion: print w/replies, xml ) Need Help??

Eye beauty has asked for the wisdom of the Perl Monks concerning the following question:

Hi do any body know how to compare two text files and find the different lines and print in in another txt file......The problem is the size of the file is large(About 800Mb to 2Gb)...........I tried the Algorithm::Diff module but for that v have to give two big arrays which will takes lots of memory......So please help me......

Replies are listed 'Best First'.
Re: Text difference
by erroneousBollock (Curate) on Nov 21, 2007 at 13:58 UTC
    The problem is the size of the file is large(About 800Mb to 2Gb)
    Repeating what was discussed in the CB, tye maintains Algorithm::Diff so he'd be your best bet for issues related to the module.

    With such large files though, you may need to change your approach. Assuming that Algorithm::Diff is bogging down due to the large file size, the idea is that you should make sure that not so much of the data is loaded at one time.

    Method 1

    Algorithm::Diff works from two arrays, right? If so perhaps the easiest thing to do is just to pass it (references to) two Tie::File objects.

    Update: well, it's not that simple. Algorithm::Diff builds a hash to keep track of indexes, which grows (O(?)) with the length of the arrays passed.

    Method 2

    If that didn't work (or was too slow) I'd do something like this (pseudocode):

    declare @differences open file1 and file2 read chunk1 from file1 (64mbyte) read chunk2 from file2 (64mbyte) while (chunk1 contains data) compute h1: md5hex of chunk1 compute h2: md5hex of chunk2 if h1 != h2 compute @diff: pass chunk1 and chunk2 to Algorithm::Diff append @diff to @differences end-of-if read chunk1 from file1 (64mbyte) read chunk2 from file2 (64mbyte) end-of-while close file1 and file2 #maybe combine adjacent deltas in @differences (eg: by line number) report @differences
    That way Algorithm::Diff only ever deals with 2x 64Mbyte of data at a time. If two adjacent chunks both contain differences between the files, you might investigate whether there's some way to combine those differences.

    -David

Re: Text difference
by KurtSchwind (Chaplain) on Nov 21, 2007 at 12:59 UTC
    If you are in unix-land, you can just use the diff command outright. It's a lot lighter on the mem requirement.
    diff a.txt b.txt > c.txt
    --
    I used to drive a Heisenbergmobile, but every time I looked at the speedometer, I got lost.
Re: Text difference
by Aim9b (Monk) on Nov 21, 2007 at 19:29 UTC
    If this is a regular occurrance you probably want a programatic solution, but if not, & you're in Windows, Visual SlickEDIT has a GUI DIFF function that I use quite often. It can handle large files, and they have a free trial downloadable.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://652114]
Approved by marto
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (8)
As of 2024-03-28 11:29 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found