Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi! I'm writing a web application and I'd like to add proxy support in case the user wants anonymity or has to use a proxy to access the web.

LWP::UserAgent claims to support proxies, but I've had limited success. I can go through one proxy without problems: 

use LWP::UserAgent;

$ua = LWP::UserAgent->new;

$ua->proxy("http", "http://proxy_host:proxy_port/");
    
my $req = HTTP::Request->new(GET => "http://www.google.com/");
my $res = $ua->request($req);

if ($res->is_success) {
    print $res->content;
} else {
     print "Bad luck this time\n";
}

[download]
But how can I chain multiple proxies? I've tried: 
$ua->proxy("http", "http://proxy1_host:proxy1_port/ http://proxy2_host
+:proxy2_port/");

$ua->proxy("http", "http://proxy1_host:proxy1_port/"); 
$ua->proxy("http", "http://proxy2_host:proxy2_port/");

[download]
The later only uses proxy2 and the former only uses proxy1!

In Internet Explorer, I can easily use multiple proxies using a syntax similar to my first attempt, so it seems that is the standard method of chaining proxies. Am I missing something obvious? How can LWP::UserAgent be told to chain proxies?

Should I use another module? I looked on CPAN and at the HTTP::Lite docs, but it from what I gather, this module only supports a single proxy too.

I'd really like to have proxy chaining support; do no Perl modules support it? Shall I roll my own HTTP implementation??

Replies are listed 'Best First'.
•Re: Chaining proxies with LWP::UserAgent
by merlyn (Sage) on Jun 28, 2003 at 08:27 UTC
    There's nothing in the HTTP protocol that even permits this. If IE is letting you say that, it's not able to deliver on it. So, it doesn't matter how many times you rewrite the HTTP implementation... there's no point. There's no character sequence that does this.

    -- Randal L. Schwartz, Perl hacker
    Be sure to read my standard disclaimer if this is a reply.

[reply]
      Thanks for the quick reply. I was mistaken about MSIE's capabilities, my apologies.

      However, HTTP does indeed permit this, via the CONNECT method as described in RFC 2616:

      This specification reserves the method name CONNECT for use with a proxy that can dynamically switch to being a tunnel (e.g. SSL tunneling 44).

      RFC 2817 details usage of the CONNECT method:

      5.2 Requesting a Tunnel with CONNECT

      A CONNECT method requests that a proxy establish a tunnel connection on its behalf. The Request-URI portion of the Request-Line is always an 'authority' as defined by URI Generic Syntax 2, which is to say the host name and port number destination of the requested connection separated by a colon:

      CONNECT server.example.com:80 HTTP/1.1
      Host: server.example.com:80

      To "chain" two proxies, one simply sends a CONNECT to the first proxy, requesting to connect to the second proxy. Once the tunnel is established, normal GET requests with a fully-qualified URI can be used. This works.

      I know HTTP, I just want to coax Perl into speaking this portion of the specification. Can I, using the standard modules? I searched LWP documentation for any mention of the CONNECT method, but couldn't find any occurances... are there any other HTTP modules which fully support the HTTP/1.1 spec, or do I have to write my own?

[reply]
        As the documentation for LWP says, the best place to discuss exact details for complex stuff is libwww@perl.org. However browsing the documentation and code it looks like there is an assumption built into the code that all requests will be mediated through only one level of proxy, and which proxy to use will be a function of the communication scheme used. (In other words it assumes a configuration which matches any browser you have ever seen.)

        But if use the full interface, your LWP::UserAgent object has methods named send_request(), simple_request() and just request() which send off requests with various levels of preparation and munging first. But the important thing is that they can be anything that you want.

        So I would suggest playing with using no proxy, and then start sending your own CONNECT requests, and see whether you can get it to send the correct sequence to chain levels of proxying.

        PS I have only seen multiple levels of proxying used by people who were attempting to use various open proxies to anonymize themselves. Having looked at the traffic that passed through one such proxy, the users paid attention to how much the server reported to others, and didn't seem to realize that the server they are proxying off of can keep logs including the information not passed on, and do things like hand it over to law enforcement... (Yes, I do know of at least one case where law enforcement took full advantage of this.)

[reply]

Back to Seekers of Perl Wisdom