in reply to Re^3: HTTP response: 400 Bad Request
in thread HTTP response: 400 Bad Request

but try first to set the agent string as they suggest, modify this to reflect yours: User-Agent: Sample Company Name AdminContact@<sample company domain>.com . They say "does not allow" and "managed to ensure..." so you may have a chance to do it by the book.

Either way, this is how you set the agent string with File::Fetch:

use File::Fetch;
$File::Fetch::USER_AGENT = 'abc';

my $ff = File::Fetch->new(uri => 'https://dnschecker.org/user-agent-in
+fo.php');
$ff->fetch(to=>'./abc');

[download]

bw, bliako

Replies are listed 'Best First'.
Re^5: HTTP response: 400 Bad Request
by justin423 (Scribe) on Jul 14, 2023 at 09:17 UTC
    well, this is interesting.

    this:

    $File::Fetch::USER_AGENT = 'COMPANYNAME validemail@validemail.com';

    came back as this

    Fetch failed! HTTP response: 400 Bad Request 400 Bad Request at useragent.pl line 8.

    then changing it to this:

    $File::Fetch::USER_AGENT = 'User-Agent: COMPANY validemail@validemail.com';

    Fetch failed! HTTP response: 403 Forbidden 403 Forbidden at useragent.pl line 8.

[reply]
      OK, almost there.. This worked under LWP.

      $ua->default_header('Accept-Encoding' => scalar HTTP::Message::decodable());

      $ua->default_header( USER_AGENT =>

      so now all I have to figure out is how to move those variables over to File:Fetch.

[reply]

        Firstly, set $File::Fetch::DEBUG=1;.

        Ideally, File::Fetch should have had a method to provide it with a user-created LWP object in order for the user to have full control over it. Diagonally looking at the source:

            ### set up the useragent object
    my $ua = LWP::UserAgent->new();
    $ua->timeout( $TIMEOUT ) if $TIMEOUT;
    $ua->agent( $USER_AGENT );
    $ua->from( $FROM_EMAIL );

        [download]

        It looks to me you are out of luck with File::Fetch if you want to fine-tune LWP. Though, LWP is not the only tool it uses to fetch files but it seems to me it creates them on the fly during each fetch() and with a minimal set of parameters decided by it.

        In the past I have used URI::Fetch which optionally accepts a pre-cooked user-specified LWP object which you can tune to your heart's content:

        my $ua = LWP::UserAgent->new(...);
URI::Fetch->fetch($uri, 'UserAgent' => $ua) };

        [download]

        FInally, LWP::UserAgent can be made super verbose (dumping the headers for example) with:

        use LWP::UserAgent;
use LWP::ConsoleLogger::Easy qw/debug_ua/;
my $ua = LWP::UserAgent->new(...);
debug_ua($ua, 10);

        [download]

        bw, bliako

[reply]
[d/l]
[select]
        so now all I have to figure out is how to move those variables over to File:Fetch

        Is there some reason why you are wedded to File::Fetch? If all you are doing is downloading from a URL and saving to a file then LWP::UserAgent can do that with the mirror method.

        🦛

[reply]

In Section Seekers of Perl Wisdom