Your task is not at all straightforward. JPEG images are not stored in the RGB colorspace, but in the YUV colorspace. But that's not the difficulty. The encoding is akin to storing frequencies, which means that you're likely to have lots of slight variations even in regions that look to be solidly one color.

The only module I know of that might be helpful is Image::Magick.