After digging into this with tcpdump, the issue is as follows. The _lwp_fetch method produces a request header such as the one below:

    GET /path/to/file.ext HTTP/1.1
    TE: deflate,gzip;q=0.3
    Connection: TE, close
    Authorization: Basic YW5vbnltb3VzOkZpbGUtRmV0Y2hAZXhhbXBsZS5jb20=
    From: File-Fetch@example.com
    Host: my.domain.com
    If-Modified-Since: Wed, 07 Jun 2017 03:40:39 GMT
    User-Agent: File::Fetch/0.48

[download]

While some of those headers are questionable (ie the From) for an HTTP request, the one that causes this issue is the Authorization. Decoding that Basic Auth from Base64 reveals that is:

    $ decode_base64("YW5vbnltb3VzOkZpbGUtRmV0Y2hAZXhhbXBsZS5jb20=")
    anonymous:File-Fetch@example.com

[download]

Looking into the File::Fetch source, we find this:

    if ($self->userinfo) {
        $uri->userinfo($self->userinfo);
    } elsif ($self->scheme ne 'file') {
        $uri->userinfo("anonymous:$FROM_EMAIL");
    }

[download]

(lines 593-597 of File::Fetch)

When the server gets a basic auth when it wasn't expecting one, at least for a subset of servers, it returns a 400 Bad Request error. I'm not sure the intention of putting a default basic auth in the _lwp_fetch when none was specified and when no other request method does this, but this is the reason it doesn't work, and why you need to either blacklist lwp or turn off warnings. Hope this helps those who come along after.

In reply to Re^3: HTTP response: 400 Bad Request by DennisJBell
in thread HTTP response: 400 Bad Request by JoeJohnston

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.