I don't see how you could avoid walking through the pixels, but of course, you don't have to examine them all.

For each of the 4 sides, you only walk the lines inward from the edge, until you bump into the first "color-to-keep" pixel, then you go to the next side.