Update: I wouldn't put money on the server sending back a 404 for a missing image, although any half-decent webmaster will do this (in addition to possibly supplying an error document), but I think you could rely on it for most applications. If you find this is not the case, I would think testing the mime type of the returned document would be more efficient than a HEAD request potentially followed by a GET.

410 (Gone) is another valid "this doesn't exist" response, and 403 might also be important. You might as well just detect any error return--the design might change in the future, after all. (HTTP errors start with a 4 or 5.)

=cut
--Brent Dax
There is no sig.

To be really safe here, you might check for a success code (I think it's 202, but don't " me) rather than trying to think of all the possible error codes you might get.

Also, just to throw another wrench in, just checking MIME type might not be good enough. What if the error "page" you get back is itself actually a jpeg image? I've seen that before. Don't know a way around it unless you are also checking the error code (and the webmaster configured the server sensibly to send back error codes).

--DrWhy

"If God had meant for us to think for ourselves he would have given us brains. Oh, wait..."

I would definitely check is_error first, but as mentioned before many times a missing file won't properly return a 404 code. Normally this is because the webmaster put a full URL in the ErrorDocument directive, which causes Apache to send a 302 response. What I've done before iss checked the file size. I could be fairly confideent that the images would be 80K or more, while an error page isn't likey to be more than 10K-20K. So anything under 50K is assumed an error, anything over 50K is assumed to be the image.

You could use LWP::UserAgent's get method (my $response = $ua->get('http://...')), and check $response->is_success. In case the web server incorrectly returns success, you could also try checking the content type ($response->content_type) against m/^image\//.

(...) but in reality it's grabbing the 404 message from the website.

My question is, what's the easiest way to check for the case when the URL "www.website.com/42/1.jpg" is sending back an HTML "sorry" page, not a JPG?

I'm just using LWP::Simple at the moment with getstore().

Just check the return value of getstore(), it'll simply tell you whether the fetch has failed, or whether it saved anything:

    getstore($url, $file)
       Gets a document identified by a URL and stores it in the file. The
       return value is the HTTP response code.

If the files did indeed get saved and they're HTML pages with error messages, then the webserver is misbehaving, and it returned a "200 OK" status message regardless. No other way of checking the webserver's status you can think of, would make a difference.

You can still snoop the saved file, for example using the command line utility file on Unixy systems — perhaps Cygwin has it ported to Windows.

If this is for Windows, and you can't find file, but you do have ImageMagick installed, then its command line utility identify can recognize whether you do have valid JPEG files.

Perl implementation of the file utility available from the Perl Power Tools project.

my $res_code =  mirror('http://www.yadaa.com/file.jpg', './file.jgp');
[download]

Is there any kind of mod_perl handler I can create to prevent a user from ripping my sequentially named images?

Thanks.

When I do stuff similar to this I usually do it in a one-liner so I tend to want to keep it short. I use different approaches depending on the situation, but sometimes I just keep a simple log (like print getstore($url), " $url";) which I then can use to filter away bad files (I don't want to have to download the same files twice). If you for some reason don't care for File::Type then a simple way I've used is the -T/-B file test functions to see if it's a text or binary file. If I want images only I check for -B success.

If you have any favourite command line website downloader you can easily generate an page indexing all the other pages or images you want to download and let the program do the job for you.

Hope this helps,
ihb

Forgot to mention that a very easy way to download URLs with numbers in them is with the "curl" utility.


curl -O http://www.website.com/images[1-4]/pic[001-250].jpg
[download]

will download images1/pic001.jpg, images1/pic002.jpg etc up to images4/pic250.jpg. Will also handle cookies etc for you if you want it to.