Your specific issue is that you're using the <=> operator to compare non-numeric values. Try using cmp instead. Test program to illustrate:

use strict;

my @dirs = (qw/11-08-2004 11-10-2004 11-07-2004 11-09-2004/);

print "Using <=>:\n", join("\n", sort {$b <=> $a} @dirs), "\n\n";
print "Using cmp:\n", join("\n", sort {$b cmp $a} @dirs), "\n";
[download]

Using <=>:
11-09-2004
11-07-2004
11-10-2004
11-08-2004

Using cmp:
11-10-2004
11-09-2004
11-08-2004
11-07-2004
[download]

Your directories will lexically sort as is, but they are not numbers since they have dashes.