Grab a page using openSSL

CiceroLove has asked for the wisdom of the Perl Monks concerning the following question:

I am just wanting to connect to a site via https, download the page, parse through it, and reupload it securely. As it stands, My problem is three-fold: 1) Everyone and their monk have told me they do it all the time, 2) I cannot for the life of me see where I am any different from any other code I have seen on the net, and 3) I get a repsonse from snippet one that I don't understand. I am using (or have used) openSSL 0.9.3-0.9.6 as well as all current version of SSLeay.pm and LWP. Your help in bringing a young acolyte to inner harmony and joy will be greatly rewarded in the afterlife. Both snippets begin with:

use LWP;
use LWP::UserAgent;
use LWP::Protocol::https;
[download]

and then:

my $connect = new LWP::UserAgent;
my $request = new HTTP::Request 'POST', 'https://secure.directnic.com/
+myaccount/myaccount.php';
my $response = $connect->request($request);

print $response;
[download]

This prints HTTP::Response=HASH(nxnnnennn) where n is some number. The second snippet changes

print $response;
[download]

to:

print $response->content;
print $response->as_string;
[download]

and gives me 500 (Internal Server Error) cannot connect to secure.directnic.com:443. I know that I can conect to it when I telnet to that port. Any suggestions from anyone?

Comment on Grab a page using openSSL Select or Download Code

Replies are listed 'Best First'.
Re (tilly) 1: Grab a page using openSSL by tilly (Archbishop) on Jan 25, 2001 at 07:14 UTC
I am going to go out on a limb and guess that your problem is that you do not have your proxy server set correctly for the https protocol and you are being blocked by a firewall. On Unix try this from the shell first: `export https_proxy=http://your.proxy.machine:8080/` [download] where the data you put there is appropriate for whatever your local proxies are. And then try the following (switch quotes on Windows): `perl -e 'use LWP::Simple; getprint("https://www.safeweb.com/");'` [download] Hopefully that will work. (If not then you have a very short script that should work to play with.)	[reply] [d/l] [select]
Re: Grab a page using openSSL by bsb (Priest) on Jan 25, 2001 at 08:46 UTC
LWP::Debug might give you some more details of what is going on. `use LWP::Debug qw(+);` [download] The LWP GET command line util can help: `GET -eSU url` I can't see any probs in your code, you need to be sure that it's not somewhere else (the proxy suggested earlier) Brad	[reply] [d/l] [select]
Re: Re: Grab a page using openSSL by CiceroLove (Monk) on Jan 26, 2001 at 02:17 UTC
Proxy settings seem correct as I get a 404 when I change them and a 500 when they are reverted. However, the LWP::Debug was a goldmine. When run I get, `LWP::UserAgent::new() LWP::UserAgent::request: () LWP::UserAgent::simple_request: POST url LWP::UserAgent::_need_proxy: Not proxied LWP::Protocol::http::request: ()` [download] When I try and change the last line to something like `LWP::Protocol::https::request 'POST', 'url';` [download] in my script it tells me that it can't locate object method "new" via package "LWP::Protocol::https::Request". Does this mean that the https request method is not implemented? Thanks so much for your continued help and guidance.	[reply] [d/l] [select]