I was tweaking a rather complicated data processing script of mine last week when I ran accross an output pipe I was using to massage the data:

my $output_pipe = "|sort -n -t, -k3,4 | uniq > $tmpfile";
open(OUTFILE,$output_pipe) || die "could not open file $output_pipe fo
+r output: $!";
[download]

I said to myself: "self, why are you using two processes (sort and uniq) when one (sort with -u flag) would work? That's silly." So, even though it had nothing to do with what I was tweaking, I changed the output pipe ot be:

my $output_pipe = "|sort -n -t, -k3,4 -u > $tmpfile";
[download]

and thought to myself "hah! got rid of one extra step."

Now, yesterday I noticed that the script was producing erroneous output. Since the input data often changes (depending on who produced it!) I have no formal spec for the input and I'm often tweaking the processing script to adapt it to new input formats (be liberal in what you accept). I thought that must have happened so I spent a good part of the day trying to track down what had changed. Since the data is several hundred MBs in size this process took a while.

Once I finally exhausted all possiblity that the input data was bad I turned to the script and finally I decided to test to see if "sort -u" was really equivalent to "sort | uniq"

my test data:

$ cat test.dat
1,1
1,2
1,3
2,1
2,4
2,2
2,1
1,1
[download]

$ sort -n -t, -k1,2 test.dat | uniq
1,1
1,2
1,3
2,1
2,2
2,4

That's good.
[download]

$ sort -n -t, -k1,2 -u test.dat

1,1
2,1

Not what I expected.
[download]

Aha! With multiple keys, sort -u does not behave as I had assumed.

The moral of this little story: If it's not broken, then don't mess with it! (Oh, and don't assume either!)

Update: In light of the excellent comments and feedback here, I think I'll modify the moral of this story:

1. Don't assume. Ever.
2. Make sure you have unit tests.
3. Run your unit tests. Frequently.
4. Don't change something unless you have a good reason to.
5. If you do change something, see rules 1,2,3,4 above