Firstly, set $File::Fetch::DEBUG=1;.

Ideally, File::Fetch should have had a method to provide it with a user-created LWP object in order for the user to have full control over it. Diagonally looking at the source:

    ### set up the useragent object
    my $ua = LWP::UserAgent->new();
    $ua->timeout( $TIMEOUT ) if $TIMEOUT;
    $ua->agent( $USER_AGENT );
    $ua->from( $FROM_EMAIL );
[download]

It looks to me you are out of luck with File::Fetch if you want to fine-tune LWP. Though, LWP is not the only tool it uses to fetch files but it seems to me it creates them on the fly during each fetch() and with a minimal set of parameters decided by it.

In the past I have used URI::Fetch which optionally accepts a pre-cooked user-specified LWP object which you can tune to your heart's content:

my $ua = LWP::UserAgent->new(...);
URI::Fetch->fetch($uri, 'UserAgent' => $ua) };
[download]

FInally, LWP::UserAgent can be made super verbose (dumping the headers for example) with:

use LWP::UserAgent;
use LWP::ConsoleLogger::Easy qw/debug_ua/;
my $ua = LWP::UserAgent->new(...);
debug_ua($ua, 10);
[download]

bw, bliako

surprise #2...

cpan> install URI::Fetch

Warning: Cannot install URI::Fetch, don't know what it is.

nope, something wrong with your installation

so now all I have to figure out is how to move those variables over to File:Fetch

Is there some reason why you are wedded to File::Fetch? If all you are doing is downloading from a URL and saving to a file then LWP::UserAgent can do that with the mirror method.

I am not wedded to file fetch, but more familiar with it. But using this code turned a text file into binary...

$filename='/temp/edgar/workfile.txt';

$url='https://www.sec.gov/Archives/edgar/daily-index/2023/QTR3/form.20230712.idx';

my $ua = LWP::UserAgent->new(timeout => 10);

$ua->default_header('Accept-Encoding' => scalar HTTP::Message::decodable());

$ua->default_header( USER_AGENT =>'COMPANY email@example.com' );

my $res = $ua->mirror( $url, $filename );

Well, if you are going to set

$ua->default_header('Accept-Encoding' => scalar HTTP::Message::decodab
+le());
[download]

then obviously you are subsequently going to have to do the necessary decoding if you want decoded data saved in the file. You will probably find it is merely compressed. Perhaps have a look at LWP::UserAgent::Patch::FilterMirror.

---

🦛