An encrypted or compressed file will contain uniformly distributed characters in the range 0..255. You don't mention what character set or binary format make up the data, but whatever it is it will likely have enough structure to obey very different statistics. A linear scan through the file for character frequency should provide enough information for a decision.

After Compline,
Zaxo