HTTPS with lwp problem.

RedGrinGo has asked for the wisdom of the Perl Monks concerning the following question:

Hello all,

I am try to get html code from a site with a prefix of "https://ssl..." my goal is to fetch data to txt file but this site got authentication. I need to authen this site and then send a request with search terms to get the data I need. I got several error that need attention. 1. Peer certificate not verified 2. HTTP Error 411. The request must be chunked or have a content length.

    use strict;
    use warnings;
    use LWP::UserAgent;
    use Crypt::SSLeay;

    my $base = 'https://ssl...';
    my $cgi  = 'terms with authen values'

    my $wrapper = new wrapper;
    my %hash = (
             cgi_method  => 'GET',
             url        => $base,
             redirection=> 'yes',
             proxy_type => $proxy_type,
             proxy_address => $proxy_address,
             proxy_credentials => $proxy_credentials,
            );
$wrapper->call_httpd_exec(\%hash);

my $response = $hash{response};

%hash = (
             cgi_method   => 'POST',
             #cgi_param    => $cgi,
             url          => $base,
             port         => '443',
             redirection  => 'yes',
             proxy_type   => $proxy_type,
             proxy_address => $proxy_address,
             proxy_credentials => $proxy_credentials,
            );

$wrapper->call_httpd_exec(\%hash);

print %hash;

$response = $hash{response};
print "\n##/n$response##\n";
my $redirect = $hash{request}->request->uri;
print "\n##/n$redirect##\n";

%hash = (
             cgi_method   => 'GET',
             #cgi_param    => $cgi,
             url          => $redirect,
             port         => '443',
             redirection  => 'yes',
             proxy_type   => $proxy_type,
             proxy_address => $proxy_address,
             proxy_credentials => $proxy_credentials,
            );
$wrapper->call_httpd_exec(\%hash);
$response = $hash{response};

print "\n##/n$response##\n";
[download]

any help will be more than welcom. Thanks a lot Monks

Comment on HTTPS with lwp problem. Download Code

Replies are listed 'Best First'.
Re: HTTPS with lwp problem. by ikegami (Patriarch) on Nov 03, 2009 at 06:36 UTC
If you have a problem building a request the server will accept, perhaps the code you should be posting is the code where you build the request? By the way, field `cgi_method` appears to be misnamed. The request method pertains to HTTP and HTTPS, not CGI. Also aside from the question asked, '`yes`' is a poor choice for the value of the `redirection` flag. It implies '`no`' should be used to avoid redirection. (Please tell me it's not!)	[reply] [d/l] [select]
Re^2: HTTPS with lwp problem. by RedGrinGo (Novice) on Nov 03, 2009 at 07:02 UTC
this my first time around here, can you please tell me what to post? the request ?? `my $user_agent = LWP::UserAgent->new; $user_agent->protocols_allowed(['http','https']); push @{$user_agent->requests_redirectable}, 'POST'; $user_agent->cookie_jar(HTTP::Cookies->new()); my $urlObj = URI->new($url); $reqObj = new HTTP::Request($method,$urlObj,$hdrObj,$cgi_param); $resObj = $user_agent->request($reqObj);` [download] This is the request. Thankyou	[reply] [d/l]
Re^3: HTTPS with lwp problem. by ikegami (Patriarch) on Nov 03, 2009 at 08:12 UTC
Which of the three problems do you think is caused by that code? can you please tell me what to post? Ideally, a minimal piece of code that demonstrates each problem (preferably something runnable), accompanied by both the output you get and the output you expect. That's not always necessary, but you haven't given us much to go on. So far, you said you've got three errors, and you haven't shown how you've gotten any of them. What you need to post is how you got each of those errors so that we can fix that "how".	[reply]