Proper way to fetch html from a https site with PERL

tollieman has asked for the wisdom of the Perl Monks concerning the following question:

I would like to access a https: url with PERL to retreive the source from that site. What is the proper way to do this? I have used the following:

#!/usr/bin/perl

use strict;
use warnings;
use LWP::Simple;

checkit();

sub checkit {
    my $url = 'https://url.that.i.want';
    my $content = get($url);
    my @data = split (/\n/, $content);
    foreach my $line (@data) {
                print "$line";
        }

}
[download]

I get the following error: 501 protocol scheme 'https' is not supported. along with the url that i requested printed to the screen. Any suggestions? Thanks

Comment on Proper way to fetch html from a https site with PERL Download Code

Replies are listed 'Best First'.
Re: Proper way to fetch html from a https site with PERL by mkmcconn (Chaplain) on Nov 21, 2001 at 09:46 UTC
You'll need https support added to the script, as with, for example Crypt::SSLeay mkmcconn	[reply]
Re: Proper way to fetch html from a https site with PERL by blakem (Monsignor) on Nov 21, 2001 at 09:48 UTC
The encryption used in https isn't in the LWP modules... You'll need to install something like Net::SSL or Crypt::SSLeay which can work with the LWP modules to help you fetch https pages. -Blake	[reply]
Re: Proper way to fetch html from a https site with PERL by dws (Chancellor) on Nov 21, 2001 at 10:55 UTC
If you are using ActiveState Perl, `C:\> ppm install Crypt-SSLeay` Will install what you need to add https: support to LWP. It describes itself as: `Crypt-SSLeay [0.17.1] OpenSSL and SSLeay glue that provides LWP https support` Works for me.	[reply] [d/l] [select]
Re: Proper way to fetch html from a https site with PERL by kwoff (Friar) on Nov 22, 2001 at 03:16 UTC
The following isn't necessarily the "proper way", but it's something that came to mind. I find it useful when debugging Perl CGI stuff for https:// servers to use `stunnel`. It's like telnet but uses SSL protocol. Example: `$ stunnel -cr foo.com:443 GET /cgi-bin/store.pl HTTP/1.0 HTTP/1.1 302 Moved Date: Wed, 21 Nov 2001 21:32:06 GMT Server: Apache/1.3.22 (Unix) mod_ssl/2.8.5 OpenSSL/0.9.6b location: /store/index.html Connection: close Content-Type: text/plain` [download] There I used it like a telnet client. You can also pipe stuff to it (actually, you can pipe stuff to telnet, too). I'm not sure offhand how to get LWP::Simple to use stunnel, though.	[reply] [d/l]