Adding a 'Host' request header leads to unexpected LWP::UserAgent behaviour

tomgracey has asked for the wisdom of the Perl Monks concerning the following question:

Hello all. Consider the following code:

#!/usr/bin/perl

use strict;
use warnings;
use LWP::UserAgent;

my $ip = 'xxx.xxx.xxx.xxx'; # but with actual real numbers obviously
my $port = '82'; # ie not port 80
my $domain_name = 'www.example.com';


my $ua = LWP::UserAgent->new;

my $target = "http://$ip:$port/";

$ua->default_header('Host' => $domain_name);

my $response = $ua->get($target);

print "Response: ".$response->decoded_content."\n";
[download]

The 'Host' header is there so the destination server knows which website I'm after on that ip and port. Surely this header should not have any effect on where LWP sends the request? However, I'm getting this:

Response: 500 Can't connect to www.example.com:82

It seems to be taking the Host header, adding the port and going to look for that, instead of looking for what is passed to it in $target. Very odd!

If I remove the line that adds the default_header, then:

Response: 200 OK

Stumped! Can anyone shed any light?

Comment on Adding a 'Host' request header leads to unexpected LWP::UserAgent behaviour Select or Download Code

Replies are listed 'Best First'.
Re: Adding a 'Host' request header leads to unexpected LWP::UserAgent behaviour by BrowserUk (Patriarch) on Jun 16, 2016 at 16:41 UTC
It seems to be taking the Host header, adding the port and going to look for that, instead of looking for what is passed to it in $target. Very odd! Are you sure that it isn't the remote host redirecting? With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :) In the absence of evidence, opinion is indistinguishable from prejudice. Not understood.	[reply]
Re: Adding a 'Host' request header leads to unexpected LWP::UserAgent behaviour by tomgracey (Scribe) on Jun 19, 2016 at 07:39 UTC
BrowserUK - yes, you are of course correct. In my defense, this was in fact one of the first things I considered - but I now see the trap I fell into. I tested the same code with several different sites and when the behaviour was identical I assumed the message was coming from LWP. Then I started tearing my code to pieces and got all flustered. Turns out though that it wasn't a very good sample because all the sites I tested were WordPress. It seems WordPress has a bunch of hidden redirects (in canonical.php). I am now trying to figure out exactly what is going on there, and what my code is going to do in the WordPress case. There is of course nothing wrong with LWP! Thanks for your help - and sorry for the false alarm...	[reply]
Re: Adding a 'Host' request header leads to unexpected LWP::UserAgent behaviour (proof example.com) by Anonymous Monk on Jun 16, 2016 at 23:57 UTC
Stumped! Can anyone shed any light? Yeah, what BrowserUk said, websites can do whatever they want, for example, a local http echo server Read more... (973 Bytes) So you see the "host" header was added successfully and returned With the real website example.com it works exactly the same Read more... (2 kB) If you do `lwp-request -E http://google.com` you'll get a redirect to http://www.google.com If you do `lwp-request -E http://google.com -HHost:Host` google won't redirect it will just return 404 websits can do what they want	[reply] [d/l] [select]