If you don't want to implement the file format specific test for each of your extensions, then you can peek in the randomness of your data. Any well designed encryption scheme will result random looking encrypted data (to resist statistical analysis). But if the encryption is poorly designed this won't be much help for you.

Beware that this approach has serious caveats: if your data can be real/pseudo random or compressed data, then it will also look like than a pile of random bits, so for example you won't be able to distinguish between a simple and an encrypted rar/zip file. (And don't forget that simple looking document formats can use compression internally.)

For the concrete implementation search for the chi square test on CPAN (I haven't looked but I'm almost sure you'll find some implementation) and try to experiment with it whether it can be good enough for your purpose.