in reply to Re^6: HTTP response: 400 Bad Request
in thread HTTP response: 400 Bad Request

so now all I have to figure out is how to move those variables over to File:Fetch

Is there some reason why you are wedded to File::Fetch? If all you are doing is downloading from a URL and saving to a file then LWP::UserAgent can do that with the mirror method.

🦛

Replies are listed 'Best First'.
Re^8: HTTP response: 400 Bad Request
by justin423 (Scribe) on Jul 15, 2023 at 01:10 UTC
    I am not wedded to file fetch, but more familiar with it. But using this code turned a text file into binary...

    $filename='/temp/edgar/workfile.txt';

    $url='https://www.sec.gov/Archives/edgar/daily-index/2023/QTR3/form.20230712.idx';

    my $ua = LWP::UserAgent->new(timeout => 10);

    $ua->default_header('Accept-Encoding' => scalar HTTP::Message::decodable());

    $ua->default_header( USER_AGENT =>'COMPANY email@example.com' );

    my $res = $ua->mirror( $url, $filename );

[reply]

      Well, if you are going to set

      $ua->default_header('Accept-Encoding' => scalar HTTP::Message::decodab
+le());

      [download]

      then obviously you are subsequently going to have to do the necessary decoding if you want decoded data saved in the file. You will probably find it is merely compressed. Perhaps have a look at LWP::UserAgent::Patch::FilterMirror.

      🦛

[reply]
[d/l]

        never had any idea downloading plain text would be so difficult..

        This is what finally worked....

        use LWP::UserAgent ();
use parent 'HTTP::Message';
$mess = HTTP::Message->new();
$mess->encode(gzip,deflate);
$filename='/temp/edgar/workfile.txt';
$url='https://www.sec.gov/Archives/edgar/daily-index/2023/QTR3/form.20
+230712.idx';
my $ua = LWP::UserAgent->new(timeout => 10);
$ua->default_header('Accept-Encoding' =>$mess = HTTP::Message->new());
$ua->default_header( USER_AGENT =>'COMPANY youremail@email.com' );
my $res = $ua->mirror( $url, $filename );

        [download]
[reply]
[d/l]

In Section Seekers of Perl Wisdom