I am not a computer vision expert. Here's how I would do it:

Load some representative target images into Mathematica or similar program and try out various filters. Start with transformations related to edge-detection or finding continous areas of colour and brightness. Visit http://mathematica.stackexchange.com/questions/tagged/image-processing if you need help.

When you have found a filter or combination of filters that correctly recognise to a high confidence whether an image is suitable for JPEG compression or not, recreate them in Perl. Binding to various imaging toolkits written in C are available, so this shouldn't be too complicated. The filter (pipeline) simply returns a boolean, use PNG or JPEG output format accordingly.