According to your locale, a byte with value 97 should be sorted before a byte with value 66.
That's fine if you're sorting iso-latin-1 text. It means "a" will be sorted earlier than "B".
That's not fine if you're sorting numbers in their native format.

@a = map unpack('N', $_),
     sort
     map pack('N', $_),
     65, 66, 67, 97, 98, 99;

  => 97, 65, 98, 66, 99, 67  using "use locale" on your system.
     a   A   b   B   c   C

  => 65, 66, 67, 97, 98, 99  using "no locale".
     A   B   C   a   b   c
[download]

So yeah, you'd have to turn it off when it's comparing bytes representing numbers. But since you want it on for when it's comparing bytes representing text, you have a problem.