Re^2: HTTP response: 400 Bad Request

Sorry, I can’t explain why the File::Fetch module thinks (incorrectly) that the fetch operation has failed.

It's because _lwp_fetch (one of the http fetch methods) failed, presumably due to the User Agent. You can verify this by adding $File::Fetch::DEBUG = 1; before the calls to fetch.

JoeJohnston, you can replace $File::Fetch::WARN = 0; with $File::Fetch::BLACKLIST = ['lwp'];.

Comment on Re^2: HTTP response: 400 Bad Request Select or Download Code

Replies are listed 'Best First'.
Re^3: HTTP response: 400 Bad Request by DennisJBell (Initiate) on Jul 22, 2017 at 19:46 UTC
After digging into this with tcpdump, the issue is as follows. The _lwp_fetch method produces a request header such as the one below: `GET /path/to/file.ext HTTP/1.1 TE: deflate,gzip;q=0.3 Connection: TE, close Authorization: Basic YW5vbnltb3VzOkZpbGUtRmV0Y2hAZXhhbXBsZS5jb20= From: File-Fetch@example.com Host: my.domain.com If-Modified-Since: Wed, 07 Jun 2017 03:40:39 GMT User-Agent: File::Fetch/0.48` [download] While some of those headers are questionable (ie the From) for an HTTP request, the one that causes this issue is the Authorization. Decoding that Basic Auth from Base64 reveals that is: `$ decode_base64("YW5vbnltb3VzOkZpbGUtRmV0Y2hAZXhhbXBsZS5jb20=") anonymous:File-Fetch@example.com` [download] Looking into the File::Fetch source, we find this: `if ($self->userinfo) { $uri->userinfo($self->userinfo); } elsif ($self->scheme ne 'file') { $uri->userinfo("anonymous:$FROM_EMAIL"); }` [download] (lines 593-597 of File::Fetch) When the server gets a basic auth when it wasn't expecting one, at least for a subset of servers, it returns a 400 Bad Request error. I'm not sure the intention of putting a default basic auth in the _lwp_fetch when none was specified and when no other request method does this, but this is the reason it doesn't work, and why you need to either blacklist lwp or turn off warnings. Hope this helps those who come along after.	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^3: HTTP response: 400 Bad Request
by DennisJBell (Initiate) on Jul 22, 2017 at 19:46 UTC

After digging into this with tcpdump, the issue is as follows. The _lwp_fetch method produces a request header such as the one below:

    GET /path/to/file.ext HTTP/1.1
    TE: deflate,gzip;q=0.3
    Connection: TE, close
    Authorization: Basic YW5vbnltb3VzOkZpbGUtRmV0Y2hAZXhhbXBsZS5jb20=
    From: File-Fetch@example.com
    Host: my.domain.com
    If-Modified-Since: Wed, 07 Jun 2017 03:40:39 GMT
    User-Agent: File::Fetch/0.48
[download]

While some of those headers are questionable (ie the From) for an HTTP request, the one that causes this issue is the Authorization. Decoding that Basic Auth from Base64 reveals that is:

    $ decode_base64("YW5vbnltb3VzOkZpbGUtRmV0Y2hAZXhhbXBsZS5jb20=")
    anonymous:File-Fetch@example.com
[download]

Looking into the File::Fetch source, we find this:

    if ($self->userinfo) {
        $uri->userinfo($self->userinfo);
    } elsif ($self->scheme ne 'file') {
        $uri->userinfo("anonymous:$FROM_EMAIL");
    }
[download]

(lines 593-597 of File::Fetch)

When the server gets a basic auth when it wasn't expecting one, at least for a subset of servers, it returns a 400 Bad Request error. I'm not sure the intention of putting a default basic auth in the _lwp_fetch when none was specified and when no other request method does this, but this is the reason it doesn't work, and why you need to either blacklist lwp or turn off warnings. Hope this helps those who come along after.

[reply]
[d/l]
[select]