The SEC does not allow botnets or automated tools to crawl the site. Any request that has been identified as part of a botnet or an automated tool outside of the acceptable policy will be managed to ensure fair access for all users. Please declare your user agent in request headers: Sample Declared Bot Request Headers: User-Agent: Sample Company Name AdminContact@<sample company domain>.com Accept-Encoding: gzip, deflate Host: www.sec.gov

I believe you'll need to set a custom user agent header (with a contact email as it shows) rather than using LWP's default. If you do that I think it'll let you through (I want to say had to do something similarly once for something from the Treasury).

The cake is a lie.
The cake is a lie.
The cake is a lie.

but try first to set the agent string as they suggest, modify this to reflect yours: User-Agent: Sample Company Name AdminContact@<sample company domain>.com . They say "does not allow" and "managed to ensure..." so you may have a chance to do it by the book.

Either way, this is how you set the agent string with File::Fetch:

use File::Fetch;
$File::Fetch::USER_AGENT = 'abc';

my $ff = File::Fetch->new(uri => 'https://dnschecker.org/user-agent-in
+fo.php');
$ff->fetch(to=>'./abc');
[download]

bw, bliako

well, this is interesting.

this:

$File::Fetch::USER_AGENT = 'COMPANYNAME validemail@validemail.com';

came back as this

Fetch failed! HTTP response: 400 Bad Request 400 Bad Request at useragent.pl line 8.

then changing it to this:

$File::Fetch::USER_AGENT = 'User-Agent: COMPANY validemail@validemail.com';

Fetch failed! HTTP response: 403 Forbidden 403 Forbidden at useragent.pl line 8.