According to your locale, a byte with value 97 should be sorted before a byte with value 66.
That's fine if you're sorting iso-latin-1 text. It means "a" will be sorted earlier than "B".
That's not fine if you're sorting numbers in their native format.
@a = map unpack('N', $_),
sort
map pack('N', $_),
65, 66, 67, 97, 98, 99;
=> 97, 65, 98, 66, 99, 67 using "use locale" on your system.
a A b B c C
=> 65, 66, 67, 97, 98, 99 using "no locale".
A B C a b c
So yeah, you'd have to turn it off when it's comparing bytes representing numbers. But since you want it on for when it's comparing bytes representing text, you have a problem.