tunafish has asked for the wisdom of the Perl Monks concerning the following question:

I'm probably missing something incredibly obvious, but I simply can't get LWP to stop trying to verify certificates -- and I've read everything I could find on the topic. Here's my code:

#!/usr/bin/perl

use strict;
use LWP::UserAgent;
use IO::Socket::SSL;
use Data::Dumper;

$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
my $bot = LWP::UserAgent->new( 
    env_proxy => 1, 
    keep_alive => 1, 
    timeout => 300, 
    ssl_opts => { 
        verify_hostname => 0, 
        SSL_verify_mode => IO::Socket::SSL::SSL_VERIFY_NONE 
    },
); 
my @headers = (
    'User-Agent'        => 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.
+0) Gecko/20100101 Firefox/19.0',
    'Accept-Language'    => 'ie-ee,en;q=0.7,ru;q=0.3',
    'Accept-Encoding'    => 'gzip, deflate',
    'Accept'            => "text/html,application/xhtml+xml,applicatio
+n/xml;q=0.9,*/*;q=0.8",
    'Cache-Control'        => 'max-age=0'
);
my $uri='https://www.something.com';
my $response = $bot->get($uri,@headers);
    
print Dumper($response);

[download]

This yields the following response:

$VAR1 = bless( {
                 '_content' => 'Can\'t connect to www.something.com:44
+3 (certificate verify failed)

LWP::Protocol::https::Socket: SSL connect attempt failed error:1409008
+6:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed 
+at /usr/local/share/perl5/LWP/Protocol/http.pm line 47.
',
                 '_rc' => 500,
                 '_headers' => bless( {
                                        'client-warning' => 'Internal 
+response',
                                        'client-date' => 'Sat, 30 May 
+2015 07:38:43 GMT',
                                        'content-type' => 'text/plain'
+,
                                        '::std_case' => {
                                                          'client-warn
+ing' => 'Client-Warning',
                                                          'client-date
+' => 'Client-Date'
                                                        }
                                      }, 'HTTP::Headers' ),
                 '_msg' => 'Can\'t connect to www.something.com:443 (c
+ertificate verify failed)',
                 '_request' => bless( {
                                        '_content' => '',
                                        '_uri' => bless( do{\(my $o = 
+'https://www.something.com')}, 'URI::https' ),
                                        '_headers' => bless( {
                                                               'user-a
+gent' => 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101
+ Firefox/19.0',
                                                               'cache-
+control' => 'max-age=0',
                                                               'accept
+' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
+',
                                                               'accept
+-language' => 'ie-ee,en;q=0.7,ru;q=0.3',
                                                               'accept
+-encoding' => 'gzip, deflate'
                                                             }, 'HTTP:
+:Headers' ),
                                        '_method' => 'GET'
                                      }, 'HTTP::Request' )
               }, 'HTTP::Response' );

[download]

I had thought that the ssl_opts I set would prevent the UA from even trying to verify the certificates. Then I threw in $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; for good measure.

What am I doing wrong?

Replies are listed 'Best First'.
Re: Disable SSL certificate verification with LWP
by noxxi (Pilgrim) on May 30, 2015 at 08:29 UTC
    Since the behavior of the options you've used depend on the version of LWP::Prototocol::https and the OS you used and if you use the CPAN version or a patched version from the distribution it would be useful to provide these information.
[reply]

      OS is RHEL 6.

      Version of LWP::Protocol::https is... none. This module is not installed on my system. HUH? I would've expected this module to be automatically installed with LWP and/or to have thrown errors at me before this point. I've been connecting to secure servers, it's only the secure servers with self-signed certificates that have been causing problems.

      Installing the module has fixed the problem, but in the interests of knowledge, could someone explain what's happening?

[reply]
Re: Disable SSL certificate verification with LWP
by wrog (Friar) on May 30, 2015 at 21:35 UTC
    You can actually specify to the underlying socket library how you want it to verify certificates or not. The problem is the options are different depending on what socket library you're using. But there's a way to force it to use a particular one, and then you know:
    {
  local $NET::HTTPS::SSL_SOCKET_CLASS = 'IO::Socket::SSL';
  $ua = LWP::UserAgent->new(ssl_opts => \%ssl_options),
}

    [download]
    and for %ssl_options, I'll let you look it up in the IO::Socket::SSL docs.

    You could also make the setting of $NET::HTTPS::SSL_SOCKET_CLASS global if you don't have to play nice with other instances of LWP::UserAgent.

    I'll note that I got burned on this when at one point I needed a particular setting (to exclude SSLv2 in favor of SSLv3 to deal with sites that immediately drop the connection if you so much as mention SSLv2) that got outdated a couple years later when SSLv3 was likewise declared anathema, the default settings of IO::Socket::SSL having caught up to what was needed to survive, I would have been better off without the explicit setting... meaning if you're going to do this stuff you're going to have to track it...)

[reply]
[d/l]
[select]
Re: Disable SSL certificate verification with LWP
by Anonymous Monk on May 30, 2015 at 08:11 UTC
[reply]

Back to Seekers of Perl Wisdom