in reply to Re^8: HTTP response: 400 Bad Request
in thread HTTP response: 400 Bad Request

Well, if you are going to set

$ua->default_header('Accept-Encoding' => scalar HTTP::Message::decodab
+le());

[download]

then obviously you are subsequently going to have to do the necessary decoding if you want decoded data saved in the file. You will probably find it is merely compressed. Perhaps have a look at LWP::UserAgent::Patch::FilterMirror.

🦛

Replies are listed 'Best First'.
Re^10: HTTP response: 400 Bad Request
by justin423 (Scribe) on Jul 15, 2023 at 15:15 UTC

    never had any idea downloading plain text would be so difficult..

    This is what finally worked....

    use LWP::UserAgent ();
use parent 'HTTP::Message';
$mess = HTTP::Message->new();
$mess->encode(gzip,deflate);
$filename='/temp/edgar/workfile.txt';
$url='https://www.sec.gov/Archives/edgar/daily-index/2023/QTR3/form.20
+230712.idx';
my $ua = LWP::UserAgent->new(timeout => 10);
$ua->default_header('Accept-Encoding' =>$mess = HTTP::Message->new());
$ua->default_header( USER_AGENT =>'COMPANY youremail@email.com' );
my $res = $ua->mirror( $url, $filename );

    [download]
[reply]
[d/l]

      How about:

      #!/usr/bin/perl
use strict;
use warnings;
use feature 'say';
use Mojo::UserAgent;

my $url = 'https://www.sec.gov/Archives/edgar/daily-index/2023/QTR3/fo
+rm.20230712.idx';
my $ua  = Mojo::UserAgent->new;
$ua->transactor->name('COMPANY youremail@email.com');
say $ua->get( $url )->res->body;
# or
$ua->get( $url )->res->save_to('derp.txt');

      [download]
[reply]
[d/l]

In Section Seekers of Perl Wisdom