|Just another Perl shrine|
Re: File::Sort issuesby Somni (Friar)
|on Jul 09, 2011 at 02:49 UTC||Need Help??|
File::Sort is really not suited to parsing CSV files properly. If they are simple enough, it's possible, but CSV files rarely remain simple enough.
How to generically compare two CSV files is difficult to answer. It depends on whether or not you can read the entire file into memory, and if their fields match. The very simplest method would be to normalize your CSV files, sort them, and then diff them.
The simplest way of normalizing them is to parse them, and then spit them back out; if you do this with the same module for each (using the same options), theoretically any rows with the same values would output the same.
Normalizing with Text::CSV_XS is straightforward:
(My first pass used *ARGV, but this results in some odd diagnostics and weird edge cases.)
At this point, you simply sort the output. Field values and the header are irrelevant; you're simply trying to make all of your CSV files consistent so diff can make some sense of it.
diff -u <(csv-normalize csv1.csv | sort) <(csv-normalize csv2.csv | sort)
This is the simplest and quickest way of comparing two CSV files. It has the advantage of being able to work on relatively large CSV files quickly, but it won't work if the field layout differs between them.