Re: HTTP response: 400 Bad Request

Hello JoeJohnston,

I get similar results when I run your code. But note that the three calls to fetch do succeed — the die clauses are never entered — and the Fetch failed! ... message is actually just a warning generated during the fetch call. You can turn those warnings off as follows:

use File::Fetch;

$File::Fetch::WARN = 0;
[download]

Sorry, I can’t explain why the File::Fetch module thinks (incorrectly) that the fetch operation has failed. But I can confirm that the three files in question are downloaded correctly: in each case the file downloaded by fetch is identical to the corresponding file downloaded in my browser (Google Chrome).

Hope that helps,

Athanasius <°(((>< contra mundum Iustus alius egestas vitae, eros Piratica,

Comment on Re: HTTP response: 400 Bad Request Select or Download Code

Replies are listed 'Best First'.
Re^2: HTTP response: 400 Bad Request by Mr. Muskrat (Canon) on Feb 01, 2016 at 22:57 UTC
Sorry, I can’t explain why the File::Fetch module thinks (incorrectly) that the fetch operation has failed. It's because _lwp_fetch (one of the http fetch methods) failed, presumably due to the User Agent. You can verify this by adding `$File::Fetch::DEBUG = 1;` before the calls to `fetch`. JoeJohnston, you can replace `$File::Fetch::WARN = 0;` with `$File::Fetch::BLACKLIST = ['lwp'];`.	[reply] [d/l] [select]
Re^3: HTTP response: 400 Bad Request by DennisJBell (Initiate) on Jul 22, 2017 at 19:46 UTC
After digging into this with tcpdump, the issue is as follows. The _lwp_fetch method produces a request header such as the one below: `GET /path/to/file.ext HTTP/1.1 TE: deflate,gzip;q=0.3 Connection: TE, close Authorization: Basic YW5vbnltb3VzOkZpbGUtRmV0Y2hAZXhhbXBsZS5jb20= From: File-Fetch@example.com Host: my.domain.com If-Modified-Since: Wed, 07 Jun 2017 03:40:39 GMT User-Agent: File::Fetch/0.48` [download] While some of those headers are questionable (ie the From) for an HTTP request, the one that causes this issue is the Authorization. Decoding that Basic Auth from Base64 reveals that is: `$ decode_base64("YW5vbnltb3VzOkZpbGUtRmV0Y2hAZXhhbXBsZS5jb20=") anonymous:File-Fetch@example.com` [download] Looking into the File::Fetch source, we find this: `if ($self->userinfo) { $uri->userinfo($self->userinfo); } elsif ($self->scheme ne 'file') { $uri->userinfo("anonymous:$FROM_EMAIL"); }` [download] (lines 593-597 of File::Fetch) When the server gets a basic auth when it wasn't expecting one, at least for a subset of servers, it returns a 400 Bad Request error. I'm not sure the intention of putting a default basic auth in the _lwp_fetch when none was specified and when no other request method does this, but this is the reason it doesn't work, and why you need to either blacklist lwp or turn off warnings. Hope this helps those who come along after.	[reply] [d/l] [select]
Re^2: HTTP response: 400 Bad Request by JoeJohnston (Novice) on Feb 01, 2016 at 05:02 UTC
Thanks for confirming and for the info on turning off warnings. Best, Joe	[reply]