I maintain a module whose test suite contains binary data. I put the data in a separate data directory and list each file in the MANIFEST file. No trouble at all.

However, I had trouble with endianness. Your module should be endian-aware if it is to interpret the binary data. See Numbers endianness and Width in perlport.

Note that before perl 5.9.2 there was apparently no direct way to specify explicit interpretation (endianness) for numbers other than int and short (See the modifiers n, N, v and V to unpack). Of course, there are ways around this, e.g. to reverse the input to unpack depending on endianness.