I was tweaking a rather complicated data processing script of mine last week when I ran accross an output pipe I was using to massage the data:

my $output_pipe = "|sort -n -t, -k3,4 | uniq > $tmpfile";
open(OUTFILE,$output_pipe) || die "could not open file $output_pipe fo
+r output: $!";
[download]

I said to myself: "self, why are you using two processes (sort and uniq) when one (sort with -u flag) would work? That's silly." So, even though it had nothing to do with what I was tweaking, I changed the output pipe ot be:

my $output_pipe = "|sort -n -t, -k3,4 -u > $tmpfile";
[download]

and thought to myself "hah! got rid of one extra step."

Now, yesterday I noticed that the script was producing erroneous output. Since the input data often changes (depending on who produced it!) I have no formal spec for the input and I'm often tweaking the processing script to adapt it to new input formats (be liberal in what you accept). I thought that must have happened so I spent a good part of the day trying to track down what had changed. Since the data is several hundred MBs in size this process took a while.

Once I finally exhausted all possiblity that the input data was bad I turned to the script and finally I decided to test to see if "sort -u" was really equivalent to "sort | uniq"

my test data:

$ cat test.dat
1,1
1,2
1,3
2,1
2,4
2,2
2,1
1,1
[download]

$ sort -n -t, -k1,2 test.dat | uniq
1,1
1,2
1,3
2,1
2,2
2,4

That's good.
[download]

$ sort -n -t, -k1,2 -u test.dat

1,1
2,1

Not what I expected.
[download]

Aha! With multiple keys, sort -u does not behave as I had assumed.

The moral of this little story: If it's not broken, then don't mess with it! (Oh, and don't assume either!)

Update: In light of the excellent comments and feedback here, I think I'll modify the moral of this story:

1. Don't assume. Ever.
2. Make sure you have unit tests.
3. Run your unit tests. Frequently.
4. Don't change something unless you have a good reason to.
5. If you do change something, see rules 1,2,3,4 above

---

---

• Are you posting in the right place? Check out Where do I post X? to know for sure.
• Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:

  <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>

• Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
• Want more info? How to link or How to display code and escape characters are good places to start.