in reply to Re^7: HTTP response: 400 Bad Request
in thread HTTP response: 400 Bad Request

I am not wedded to file fetch, but more familiar with it. But using this code turned a text file into binary...

$filename='/temp/edgar/workfile.txt';

$url='https://www.sec.gov/Archives/edgar/daily-index/2023/QTR3/form.20230712.idx';

my $ua = LWP::UserAgent->new(timeout => 10);

$ua->default_header('Accept-Encoding' => scalar HTTP::Message::decodable());

$ua->default_header( USER_AGENT =>'COMPANY email@example.com' );

my $res = $ua->mirror( $url, $filename );

Replies are listed 'Best First'.
Re^9: HTTP response: 400 Bad Request
by hippo (Archbishop) on Jul 15, 2023 at 08:25 UTC

    Well, if you are going to set

    $ua->default_header('Accept-Encoding' => scalar HTTP::Message::decodab
+le());

    [download]

    then obviously you are subsequently going to have to do the necessary decoding if you want decoded data saved in the file. You will probably find it is merely compressed. Perhaps have a look at LWP::UserAgent::Patch::FilterMirror.

    🦛

[reply]
[d/l]

      never had any idea downloading plain text would be so difficult..

      This is what finally worked....

      use LWP::UserAgent ();
use parent 'HTTP::Message';
$mess = HTTP::Message->new();
$mess->encode(gzip,deflate);
$filename='/temp/edgar/workfile.txt';
$url='https://www.sec.gov/Archives/edgar/daily-index/2023/QTR3/form.20
+230712.idx';
my $ua = LWP::UserAgent->new(timeout => 10);
$ua->default_header('Accept-Encoding' =>$mess = HTTP::Message->new());
$ua->default_header( USER_AGENT =>'COMPANY youremail@email.com' );
my $res = $ua->mirror( $url, $filename );

      [download]
[reply]
[d/l]

        How about:

        #!/usr/bin/perl
use strict;
use warnings;
use feature 'say';
use Mojo::UserAgent;

my $url = 'https://www.sec.gov/Archives/edgar/daily-index/2023/QTR3/fo
+rm.20230712.idx';
my $ua  = Mojo::UserAgent->new;
$ua->transactor->name('COMPANY youremail@email.com');
say $ua->get( $url )->res->body;
# or
$ua->get( $url )->res->save_to('derp.txt');

        [download]
[reply]
[d/l]

In Section Seekers of Perl Wisdom