Need LWP::UserAgent --header help

Todd Chester has asked for the wisdom of the Perl Monks concerning the following question:

Dear Perl Monks,

Would some kind soul please tell me how I can duplicate

curl --header 'Host: dl.emsisoft.com' https://www.emsisoft.com/en/software/eek --

with LWP::UserAgent?

Many thanks,
-T

What I have so far

sub GetWebPageWithReferer ( $$$$ ) {
   # Return $Url's web page

   my $IAm     = ( caller(0) )[3];

   # Incoming:
   my $Url     = $_[0];
   my $Caller  = "$_[1]" . "$IAm";
   my $Referer  = $_[3];           # headers or referers


   # Outgoing:
   my $PageStatus = \$_[2];
   $$PageStatus   = 0;

   my $ua;
   my $response;
   my $WebPage;

   $ua = LWP::UserAgent->new;
   $ua->timeout ( MaxTime1 );
   $ua->show_progress;   # 0 = do not show; 1 = show

# It aint working yet !!!
   $response = $ua->get( $Url );
#   $response = $ua->get( $Url, 'Referer' => "$Referer" );
#   $response = $ua->get( $Url, 'Host' => "$Referer" );
   $WebPage = $response->content;
   # print STDERR "$WebPage\n";
   if ( ! $response->is_success )
      { $$PageStatus = DOWNLOAD_FAIL;
        print STDERR BOLD RED 
           "   ERROR ${Caller}.GetWebPageWithReferer: unable to read "
+, 
               $Url, RESET, "\n"; 
        return "";
      }
    else {
       return $WebPage;
    }
}

my $IAm = MyName ( ( caller(0) )[3] );
my $WebSite = "https://www.emsisoft.com/en/software/eek";
my $Referer = "dl.emsisoft.com";
$WebPage = GetWebPageWithReferer ( "$WebSite", $IAm, $PageStatus, "$Re
+ferer" );
[download]

Comment on Need LWP::UserAgent --header help Download Code

Replies are listed 'Best First'.
Re: Need LWP::UserAgent --header help by choroba (Cardinal) on Jan 29, 2017 at 23:49 UTC
First: your sample code is not a SSCCE. I had to add some use statements, and declare some variables to make it strict compliant. Examining the response as `print STDERR " ERROR ${Caller}.GetWebPageWithReferer: unable to read " +, $Url, " because ", $response->content, "\n";` [download] showed the following among others: `Access denied \| www.emsisoft.com used Cloudflare to restrict access The owner of this website (www.emsisoft.com) has banned your access ba +sed on your browser's signature (blahblahblah).` [download] I tried to supply a more friendly user agent identifier to the constructor: `$ua = LWP::UserAgent->new(agent => 'Mozilla');` [download] and voilà: it started working. It seems you might be breaking the law, though - check their Terms of use. BTW, you might need LWP::Protocol::https to access https sites. ($q=q:Sq=~/;[c](.)(.)/;chr(-\|\|-\|5+lengthSq)`"S\|oS2"`map{chr \|+ord }map{substrSq`S_+\|`\|}3E\|-\|`7**2-3:)=~y+S\|`+$1,++print+eval$q,q,a, [download]	[reply] [d/l] [select]
Re^2: Need LWP::UserAgent --header help by Todd Chester (Scribe) on Jan 30, 2017 at 04:09 UTC
$ua = LWP::UserAgent->new(agent => 'Mozilla'); did the trick. Thank you!	[reply]
Re: Need LWP::UserAgent --header help by Corion (Patriarch) on Jan 30, 2017 at 09:12 UTC
Back in the day, I wrote HTTP::Request::FromTemplate for that, but I mostly found it easier to replicate the request using plain Perl code setting the headers instead of maintaining a template for the request. But maybe the approach works better for you than it does for me.	[reply]
Re: Need LWP::UserAgent --header help by Anonymous Monk on Jan 29, 2017 at 23:47 UTC
First step is first, find out what actually gets sent, then duplicate it Read more... (905 Bytes) The referer does get sent , but its just one header, and webservers can be stupidly specific in what headers they like Read more... (972 Bytes)	[reply] [d/l] [select]