in reply to Re^4: HTTP response: 400 Bad Request
in thread HTTP response: 400 Bad Request

well, this is interesting.

this:

$File::Fetch::USER_AGENT = 'COMPANYNAME validemail@validemail.com';

came back as this

Fetch failed! HTTP response: 400 Bad Request 400 Bad Request at useragent.pl line 8.

then changing it to this:

$File::Fetch::USER_AGENT = 'User-Agent: COMPANY validemail@validemail.com';

Fetch failed! HTTP response: 403 Forbidden 403 Forbidden at useragent.pl line 8.

Replies are listed 'Best First'.
Re^6: HTTP response: 400 Bad Request
by justin423 (Scribe) on Jul 14, 2023 at 10:27 UTC
    OK, almost there.. This worked under LWP.

    $ua->default_header('Accept-Encoding' => scalar HTTP::Message::decodable());

    $ua->default_header( USER_AGENT =>

    so now all I have to figure out is how to move those variables over to File:Fetch.

[reply]

      Firstly, set $File::Fetch::DEBUG=1;.

      Ideally, File::Fetch should have had a method to provide it with a user-created LWP object in order for the user to have full control over it. Diagonally looking at the source:

          ### set up the useragent object
    my $ua = LWP::UserAgent->new();
    $ua->timeout( $TIMEOUT ) if $TIMEOUT;
    $ua->agent( $USER_AGENT );
    $ua->from( $FROM_EMAIL );

      [download]

      It looks to me you are out of luck with File::Fetch if you want to fine-tune LWP. Though, LWP is not the only tool it uses to fetch files but it seems to me it creates them on the fly during each fetch() and with a minimal set of parameters decided by it.

      In the past I have used URI::Fetch which optionally accepts a pre-cooked user-specified LWP object which you can tune to your heart's content:

      my $ua = LWP::UserAgent->new(...);
URI::Fetch->fetch($uri, 'UserAgent' => $ua) };

      [download]

      FInally, LWP::UserAgent can be made super verbose (dumping the headers for example) with:

      use LWP::UserAgent;
use LWP::ConsoleLogger::Easy qw/debug_ua/;
my $ua = LWP::UserAgent->new(...);
debug_ua($ua, 10);

      [download]

      bw, bliako

[reply]
[d/l]
[select]
        surprise #2...

        cpan> install URI::Fetch

        Warning: Cannot install URI::Fetch, don't know what it is.

[reply]
      so now all I have to figure out is how to move those variables over to File:Fetch

      Is there some reason why you are wedded to File::Fetch? If all you are doing is downloading from a URL and saving to a file then LWP::UserAgent can do that with the mirror method.

      🦛

[reply]
        I am not wedded to file fetch, but more familiar with it. But using this code turned a text file into binary...

        $filename='/temp/edgar/workfile.txt';

        $url='https://www.sec.gov/Archives/edgar/daily-index/2023/QTR3/form.20230712.idx';

        my $ua = LWP::UserAgent->new(timeout => 10);

        $ua->default_header('Accept-Encoding' => scalar HTTP::Message::decodable());

        $ua->default_header( USER_AGENT =>'COMPANY email@example.com' );

        my $res = $ua->mirror( $url, $filename );

[reply]

In Section Seekers of Perl Wisdom