HTTP Proxy Madness

digamma has asked for the wisdom of the Perl Monks concerning the following question:

UPDATE: I figured it out. It turns out asking for "http://www.google.com" is very different from asking for "http://www.google.com/". Thanks to everyone who replied. Original post is below. I understand that the answer to this question is going to be something like, "You're missing a hyphen, jackass!" But I've banged my head against the wall long enough. This script should hit my HTTP proxy, retrieve the Google front page, and display the HTTP return code and the HTML content to standard output. I think.

use strict;

use LWP::UserAgent;

our $ua = LWP::UserAgent->new;
$ua->proxy('http', "http://localhost:8080");

my $url = "http://www.google.com";

my $response = $ua->get($url);

print $response->status_line, "\n";
print $response->content, "\n";
[download]

Running it returns "400 Invalid header received from browser". I know the proxy works because both Mozilla and Wget are happy to use it. It's possible that the proxy requires broken HTTP that both of those use and LWP::UserAgent doesn't, I suppose. Any help would be much appreciated.

Comment on HTTP Proxy Madness Download Code

Replies are listed 'Best First'.
Re: HTTP Proxy Madness by ikegami (Patriarch) on Sep 20, 2004 at 17:07 UTC
`use LWP; our $ua = LWP::UserAgent->new(); my $url = "http://www.google.com"; my $response = $ua->get($url); my $request = $response->request(); print $request->as_string(); __END__ output ====== GET http://www.google.ca/ User-Agent: libwww-perl/5.64` [download] Ouch, LWP has bugs. The lack of HTTP/1.0 or HTTP/1.1 makes this an HTTP/0.9 request. That's fine, except that a HTTP/0.9 header consists of just the Request-Line, with no "header: value" pairs. The User-Agent line and the three CRLF after the first one are therefore invalid, but I doubt this is the problem. An absolute-URI is specified in the Request-Line. That "is only allowed when the request is being made to a proxy". I'm not making the request to a proxy, so bug number two! You are making the request to a proxy, however, so that should be ok for you. Should be ok. I bet this is what it's chocking on nonetheless.	[reply] [d/l]
Re: HTTP Proxy Madness by sintadil (Pilgrim) on Sep 20, 2004 at 15:33 UTC
Running it returns "400 Invalid header received from browser". Perhaps you should run a packet sniffer while Mozilla / $other_working_program do their thing, and see what those programs send in order to work. Ethereal seems pretty popular, though I prefer ngrep because it doesn't need any fancy X GUI which makes it handy to run on servers and remote machines, which is where you're most likely to need a sniffer anyway.	[reply]
Re^2: HTTP Proxy Madness by Fletch (Bishop) on Sep 20, 2004 at 15:46 UTC
Ethereal also comes with `tethereal` which will run fine from just a tty. You can also use `tethereal` (or another packet sniffer such as `tcpdump` or `snoop`) to dump traffic to a file and then pull the results back to your local machine to dissect them using the full power of the Ethereal GUI. Update: Just to clarify, `tethereal` has no X parts and will spit out most of the same information as `ethereal` can: 0.021897 199.242.139.15 -> 199.242.139.11 HTTP HEAD / HTTP/1.1 0000 00 10 4b 66 c1 93 00 e0 29 27 1a ae 08 00 45 00 ..Kf....)'.... +E. 0010 00 a8 c2 54 40 00 3f 06 d2 fb c7 f2 8b 0f c7 f2 ...T@.?....... +.. 0020 8b 0b d8 89 00 50 67 3b 94 99 0d bf 1c 41 80 18 .....Pg;.....A +.. 0030 82 18 89 7b 00 00 01 01 08 0a 07 2d 1c 04 07 2d ...{.......-.. +.- 0040 1c 02 48 45 41 44 20 2f 20 48 54 54 50 2f 31 2e ..HEAD / HTTP/ +1. 0050 31 0d 0a 54 45 3a 20 64 65 66 6c 61 74 65 2c 67 1..TE: deflate +,g 0060 7a 69 70 3b 71 3d 30 2e 33 0d 0a 43 6f 6e 6e 65 zip;q=0.3..Con +ne 0070 63 74 69 6f 6e 3a 20 54 45 2c 20 63 6c 6f 73 65 ction: TE, clo +se 0080 0d 0a 48 6f 73 74 3a 20 70 68 79 64 65 61 75 78 ..Host: phydea +ux 0090 2e 6f 72 67 0d 0a 55 73 65 72 2d 41 67 65 6e 74 .org..User-Age +nt 00a0 3a 20 6c 77 70 2d 72 65 71 75 65 73 74 2f 32 2e : lwp-request/ +2. 00b0 30 31 0d 0a 0d 0a 01.... 0.027344 199.242.139.11 -> 199.242.139.15 HTTP HTTP/1.1 200 OK 0000 00 e0 29 27 1a ae 00 10 4b 66 c1 93 08 00 45 00 ..)'....Kf.... +E. 0010 00 ec c2 55 40 00 40 06 00 00 c7 f2 8b 0b c7 f2 ...U@.@....... +.. 0020 8b 0f 00 50 d8 89 0d bf 1c 41 67 3b 95 0d 80 18 ...P.....Ag;.. +.. 0030 82 18 a6 de 00 00 01 01 08 0a 07 2d 1c 05 07 2d ...........-.. +.- 0040 1c 04 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f ..HTTP/1.1 200 + O 0050 4b 0d 0a 44 61 74 65 3a 20 4d 6f 6e 2c 20 32 30 K..Date: Mon, +20 0060 20 53 65 70 20 32 30 30 34 20 31 36 3a 32 31 3a Sep 2004 16:2 +1: 0070 34 35 20 47 4d 54 0d 0a 53 65 72 76 65 72 3a 20 45 GMT..Server +: 0080 41 70 61 63 68 65 2f 31 2e 33 2e 33 31 20 28 55 Apache/1.3.31 +(U 0090 6e 69 78 29 20 44 41 56 2f 31 2e 30 2e 33 20 6d nix) DAV/1.0.3 + m 00a0 6f 64 5f 70 65 72 6c 2f 31 2e 32 39 20 6d 6f 64 od_perl/1.29 m +od 00b0 5f 73 73 6c 2f 32 2e 38 2e 31 39 20 4f 70 65 6e _ssl/2.8.19 Op +en 00c0 53 53 4c 2f 30 2e 39 2e 36 65 0d 0a 43 6f 6e 6e SSL/0.9.6e..Co +nn 00d0 65 63 74 69 6f 6e 3a 20 63 6c 6f 73 65 0d 0a 43 ection: close. +.C 00e0 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 74 65 78 ontent-Type: t +ex 00f0 74 2f 68 74 6d 6c 0d 0a 0d 0a t/html.... [download] But you can't do things like use the "follow this connection" tool which the Ethereal GUI provides.	[reply] [d/l]
Re^3: HTTP Proxy Madness by sintadil (Pilgrim) on Sep 20, 2004 at 15:57 UTC
Okay, but correct me if I'm wrong, but you still need a GUI for that, right? Some people just don't have one available to them. I, for example, don't have one available to me right now because I'm in the process of updating my GUI software -- not that I'd use a GUI for packet sniffing, or any system administration anyway.	[reply]