Proxies with LWP & SSL

spatterson has asked for the wisdom of the Perl Monks concerning the following question:

Greetings Monks

I'm trying to use LWP::Useragent to fetch an SSL webpage via a proxy, using the following code (snipped down a bit from the actual code, but tested). This works OK for http urls but I get a '400 bad request' error for any https urls. This same proxy works fine for generic web browsers (firefox/opera etc).

Any ideas?

use LWP::UserAgent;
use LWP::Debug qw(+);

my $browser = LWP::UserAgent->new(keep_alive => 1);
$browser->cookie_jar({}); # for the session cookie
$browser->proxy(['http', 'https'], 'http://192.168.2.1:8118/');
my $url = "https://www.hsbc.co.uk/";
my $response = $browser->get($url);
print $response->status_line;
[download]

Debugging information from LWP

LWP::UserAgent::new: ()
LWP::UserAgent::proxy: ARRAY(0x224e2c) http://192.168.2.1:8118/
LWP::UserAgent::proxy: http http://192.168.2.1:8118/
LWP::UserAgent::proxy: https http://192.168.2.1:8118/
LWP::UserAgent::request: ()
HTTP::Cookies::add_cookie_header: Checking www.hsbc.co.uk for cookies
HTTP::Cookies::add_cookie_header: Checking .hsbc.co.uk for cookies
HTTP::Cookies::add_cookie_header: Checking hsbc.co.uk for cookies
HTTP::Cookies::add_cookie_header: Checking .co.uk for cookies
HTTP::Cookies::add_cookie_header: Checking co.uk for cookies
HTTP::Cookies::add_cookie_header: Checking .uk for cookies
LWP::UserAgent::send_request: GET https://www.hsbc.co.uk/
LWP::UserAgent::_need_proxy: Proxied to http://192.168.2.1:8118/
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 748 bytes
LWP::Protocol::collect: read 303 bytes
LWP::UserAgent::request: Simple response: Bad Request
[download]

just another cpan module author

Comment on Proxies with LWP & SSL Select or Download Code

Replies are listed 'Best First'.
Re: Proxies with LWP & SSL by inman (Curate) on Dec 20, 2005 at 11:37 UTC
LWP tries to use Crypt::SSLeay when you request a page using HTTPS. Crypt::SSLeay doesn't use the proxy information that you have set up in LWP. You need to set the proxy information as an environment variable instead. `use strict; use warnings; use LWP::UserAgent; use LWP::Debug qw(+); $ENV{HTTP_Proxy} = 'http://192.168.2.1:8118'; my $browser = LWP::UserAgent->new(keep_alive => 1); $browser->env_proxy(1); my $url = "https://www.hsbc.co.uk/"; my $response = $browser->get($url); print $response->as_string();` [download] As a general point though - it looks as though you are trying to screen scrape a banking application. Take a look at WWW::Mechanize as a better way of managing navigation and form filling.	[reply] [d/l]

Replies are listed 'Best First'.

Re: Proxies with LWP & SSL
by inman (Curate) on Dec 20, 2005 at 11:37 UTC

Crypt::SSLeay

use strict;
use warnings;

use LWP::UserAgent;
use LWP::Debug qw(+);

$ENV{HTTP_Proxy} = 'http://192.168.2.1:8118';

my $browser = LWP::UserAgent->new(keep_alive => 1);
$browser->env_proxy(1);
my $url = "https://www.hsbc.co.uk/";
my $response = $browser->get($url);
print $response->as_string();
[download]

As a general point though - it looks as though you are trying to screen scrape a banking application. Take a look at WWW::Mechanize as a better way of managing navigation and form filling.

[reply]
[d/l]