HTTP_REFERRER is a client side constructed entity, IIRC. Not all browsers send it, and none are required to as I understand. It is trivially spoofed, and shouldn't be counted on too heavily.

Why don't you let the image reside (and be fetched from) the server you want?