Perhaps finding or making a tool that will crop and scale in one pass while "loading" will give you most of the benifit. Then you have data for a screen-sized image, which is much smaller than the original. I beleive you need to keep three rows on hand in order to do bicubic interpolation to resample the image. For cropping, you can throw away whole blocks and not even process them; though you still have to read through it because you don't know the length ahead of time.

Look at the code ImageMagick uses to read a JPEG file. I beleive that is the independent free JPEG source.