mje has asked for the wisdom of the Perl Monks concerning the following question:

I'm migrating a script from 5.10.0 to 5.14.0 and a GET on a secure web server fails with "certificate verify failed" even though I know the site has a valid certificate:

use LWP::UserAgent; 
use strict;
use warnings;

my $ua  = LWP::UserAgent->new; 
my $req = HTTP::Request->new(GET => 'https://www.easysoft.com');
my $res = $ua->request($req);

print $res->headers_as_string;
print $res->content;

[download]

returns content fine in 5.10.0 and headers like this:

Connection: close
Date: Thu, 16 Jun 2011 14:22:46 GMT
Accept-Ranges: bytes
Server: Apache/2.0.54 (Unix) mod_ssl/2.0.54 OpenSSL/0.9.7d mod_perl/1.
+999.21 Perl/v5.8.6
Vary: Accept-Encoding
Content-Type: text/html; charset=ISO-8859-1
Client-Date: Thu, 16 Jun 2011 14:22:22 GMT
Client-Peer: 172.20.100.10:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/O=Equifax/OU=Equifax Secure Certificate 
+Authority
Client-SSL-Cert-Subject: /serialNumber=Paoxfx3blSdh6U20B0CULwa1WF0wpCX
+i/C=GB/O=www.easysoft.com/OU=GT68879435/OU=See www.rapidssl.com/resou
+rces/cps (c)10/OU=Domain Control Validated - RapidSSL(R)/CN=www.easys
+oft.com
Client-SSL-Cipher: DHE-RSA-AES256-SHA
Client-SSL-Warning: Peer certificate not verified
Client-Transfer-Encoding: chunked
Content-Style-Type: text/css

[download]

Same code in 5.14.0 returns:

Content-Type: text/plain
Client-Date: Thu, 16 Jun 2011 14:26:04 GMT
Client-Warning: Internal response
Can't connect to www.easysoft.com:443

[download]

and if I add $ENV{HTTPS_CA_FILE} = "/usr/share/ca-certificates/cacert.org/cacert.org.crt" to the script and run in 5.14.0 I get:

Content-Type: text/plain
Client-Date: Thu, 16 Jun 2011 14:26:52 GMT
Client-Warning: Internal response
Can't connect to www.easysoft.com:443 (certificate verify failed)

LWP::Protocol::https::Socket: SSL connect attempt failed with unknown 
+errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certific
+ate verify failed at /home/martin/perl5/perlbrew/perls/perl-5.14.0/li
+b/site_perl/5.14.0/LWP/Protocol/http.pm line 51.

[download]

I had a similar problem connecting to facebook which I was told would be resolved if I installed Mozilla::CA but I already had that installed. In the end I had to I copy the certificates and put them into a "certs" file then a simple "export HTTPS_CA_FILE=/home/martin/certs" made it work. Surely this is not correct.

This is just an example. I'm actually trying to connect to api.betfair.com but this has a valid certificate as well as verified in my browser but api.betfair.com does not return any content so I decided against using it in my example.

Any ideas?

UPDATE Should have mentioned perl 5.10.0 is system Perl on ubuntu and perl 5.14.0 is installed under perlbrew - just in case it makes a difference.

UPDATE2 HTTPS_DEBUG=1 produces output under 5.10.0 and nothing under 5.14.0.

UPDATE3 I had PERL_UNICODE=SAL and unsetting it fixes the problem.

Solution It appears I was missing intermediate certificate 0xeb99629b. Thanks to daxim for putting me on the right track. You can find the details at failed connect or “certificate verify failed” on LWP HTTPS GET

Replies are listed 'Best First'.
Re: "certificate verify failed" difference between Perl 5.14 and 5.10
by ikegami (Patriarch) on Jun 16, 2011 at 16:34 UTC

    Works for me.

    LWP 6.02, 5.14.0, ActiveState, x86, Windows

    LWP 6.02, 5.12.1, perlbrew, x86_64, Linux, OpenSSL-0.9.8g

    The certificate is by Equifax, and Equifax is in the .pem.

    Check that Mozilla::CA::SSL_ca_file() returns the right file and that you have permission to read the file.

[reply]
[d/l]
      $ perl -MMozilla::CA -le 'print Mozilla::CA::SSL_ca_file();'
/home/martin/perl5/perlbrew/perls/perl-5.14.0/lib/site_perl/5.14.0/Moz
+illa/CA/ca
cert.pem
$ ls -la /home/martin/perl5/perlbrew/perls/perl-5.14.0/lib/site_perl/5
+.14.0/Mozilla/CA/cacert.pem
-r--r--r-- 1 martin root 256791 2011-04-09 16:23 /home/martin/perl5/pe
+rlbrew/perls/perl-5.14.0/lib/site_perl/5.14.0/Mozilla/CA/cacert.pem
$ perl -MLWP -le 'print $LWP::VERSION;'
6.02

      [download]

      and I can cat the pem file as the user I am running the script as.

[reply]
[d/l]
        It could be a problem in the OpenSSL library??? Or maybe the .pem is corrupted??? (Line endings???)
[reply]
Re: "certificate verify failed" difference between Perl 5.14 and 5.10
by Corion (Patriarch) on Jun 16, 2011 at 14:39 UTC

    You are using LWP::UserAgent with a version above 6.0. You will need to install a certificate authority database like Mozilla::CA (the one by Mozilla).

[reply]

      As I mentioned in the post, I already have Mozilla::CA:

      perl -MMozilla::CA -le 'print $Mozilla::CA::VERSION'
20110409

      [download]
[reply]
[d/l]
Re: "certificate verify failed" difference between Perl 5.14 and 5.10
by Anonymous Monk on Jun 16, 2011 at 16:32 UTC
    You forgot to check the module versions
    $ perl -d:Modlist -S lwp-request -UusSeEd https://www.easysoft.com
GET https://www.easysoft.com
User-Agent: lwp-request/6.00 libwww-perl/6.02

200 OK
Connection: close
Date: Thu, 16 Jun 2011 16:31:20 GMT
Accept-Ranges: bytes
Server: Apache/2.0.54 (Unix) mod_ssl/2.0.54 OpenSSL/0.9.7d mod_perl/1.
+999.21 Perl/v5.8.6
Vary: Accept-Encoding
Content-Type: text/html; charset=ISO-8859-1
Client-Date: Thu, 16 Jun 2011 16:32:14 GMT
Client-Peer: 89.238.155.10:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/O=Equifax/OU=Equifax Secure Certificate 
+Authority
Client-SSL-Cert-Subject: /serialNumber=Paoxfx3blSdh6U20B0CULwa1WF0wpCX
+i/C=GB/O=www.easysoft.com/OU=GT68879435/OU=See www.rapidssl.com/resou
+rces/cps (c)10/OU=Domain Control Validated - RapidSSL(R)/CN=www.easys
+oft.com
Client-SSL-Cipher: DHE-RSA-AES256-SHA
Client-SSL-Socket-Class: IO::Socket::SSL
Client-Transfer-Encoding: chunked
Content-Style-Type: text/css
Link: </style/style.css>; rel="stylesheet"; type="text/css"
Link: </favicon.ico>; rel="icon"; type="image/x-icon"
Title: ODBC, JDBC and XML Driver Downloads for Windows, Unix, Linux an
+d Mac OS X
X-Meta-Description: Easysoft ODBC, JDBC and XML drivers let you access
+ Oracle, SQL Server, Access, InterBase, Sybase, Firebird, RMS, ISAM, 
+Coda and Linc from Windows, Unix, Linux, Mac OS X and OpenVMS.
X-Meta-Keywords: easysoft,odbc,jdbc,xml,drivers,gateway,bridge,data,da
+tabases,files,oracle,interbase,sybase,firebird,microsoft,sql server,a
+ccess,coda,openvms,rms,isam,linc
X-Meta-Verify-V1: A80XwI68Uh8cIAFxkNSpouY1b8iRx/PeYQcfrBUotks=

Use of uninitialized value in split at C:/perl/site/5.12.2/lib/Devel/M
+odlist.pm line 49.
AutoLoader             5.71
Carp                   1.17
Compress::Raw::Zlib   2.035
Config
Config_git.pl
Config_heavy.pl
Cwd                    3.33
DynaLoader             1.10
Encode                 2.43
Encode::Alias          2.14
Encode::Byte           2.04
Encode::Config         2.05
Encode::Encoding       2.05
Encode::Locale         1.02
Errno                  1.11
Exporter             5.64_01
Exporter::Heavy      5.64_01
Fcntl                  1.06
File::Basename         2.78
File::Glob             1.07
File::GlobMapper      1.000
File::Spec             3.33
File::Spec::Unix       3.33
File::Spec::Win32      3.33
FileHandle             2.02
Getopt::Long           2.38
HTML::Entities         3.68
HTML::HeadParser       3.66
HTML::Parser           3.68
HTTP::Config           6.00
HTTP::Date             6.00
HTTP::Headers          6.00
HTTP::Message          6.02
HTTP::Request          6.00
HTTP::Response         6.01
HTTP::Status           6.00
IO                   1.25_02
IO::Compress::Base::Common  2.035
IO::Compress::Gzip::Constants  2.035
IO::Compress::Zlib::Extra  2.035
IO::File               1.14
IO::Handle             1.28
IO::Seekable            1.1
IO::Socket             1.31
IO::Socket::INET       1.31
IO::Socket::SSL        1.44
IO::Socket::UNIX       1.23
IO::Uncompress::Adapter::Inflate  2.035
IO::Uncompress::Base  2.035
IO::Uncompress::Gunzip  2.035
IO::Uncompress::RawInflate  2.035
LWP                    6.02
LWP::MemberMixin
LWP::Protocol          6.00
LWP::Protocol::http
LWP::Protocol::https
LWP::UserAgent         6.02
List::Util             1.23
Mozilla::CA          20110409
Net::HTTP              6.01
Net::HTTP::Methods     6.00
Net::HTTPS             6.00
Net::IDN::Encode        1.1
Net::IDN::Nameprep      1.1
Net::IDN::Punycode    1.000
Net::IDN::Punycode::PP  1.000
Net::SSLeay            1.36
Scalar::Util           1.23
SelectSaver            1.02
Socket                 1.87
Storable               2.25
Symbol                 1.07
Time::Local          1.2000
URI                    1.58
URI::Escape            3.30
URI::Heuristic         4.19
URI::_generic
URI::_idna
URI::_punycode         0.03
URI::_query
URI::_server
URI::http
URI::https
Unicode::Normalize     1.12
Unicode::Stringprep   1.103
Unicode::Stringprep::BiDi   1.10
Unicode::Stringprep::Mapping   1.10
Unicode::Stringprep::Prohibited   1.10
Unicode::Stringprep::Unassigned   1.10
Unicode::Stringprep::_Common   1.10
Win32::API             0.62
Win32::API::Struct     0.62
Win32::API::Type       0.62
XSLoader               0.15
base                   2.15
bytes                  1.04
constant               1.21
integer                1.00
overload               1.10
unicore::Heavy.pl
unicore::To::Fold.pl
unicore::To::Lower.pl
unicore::lib::Nt::De.pl
unicore::lib::Perl::SpacePer.pl
utf8                   1.08
utf8_heavy.pl
vars                   1.01
warnings               1.09
warnings::register     1.01

    [download]
[reply]
[d/l]
      perl -d:Modlist -S lwp-request -UusSeEd https://www.easysoft.com

      [download]

      works for me i.e., it downloads the file but my starting example still does not work.

      martin@betdevel:~$ perl -d:Modlist -S lwp-request -UusSeEd https://www
+.easysoft
.com
GET https://www.easysoft.com
User-Agent: lwp-request/6.00 libwww-perl/6.02

200 OK
Connection: close
Date: Thu, 16 Jun 2011 17:05:08 GMT
Accept-Ranges: bytes
Server: Apache/2.0.54 (Unix) mod_ssl/2.0.54 OpenSSL/0.9.7d mod_perl/1.
+999.21 Per
l/v5.8.6
Vary: Accept-Encoding
Content-Type: text/html; charset=ISO-8859-1
Client-Date: Thu, 16 Jun 2011 17:04:44 GMT
Client-Peer: 172.20.100.10:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/O=Equifax/OU=Equifax Secure Certificate 
+Authority
Client-SSL-Cert-Subject: /serialNumber=Paoxfx3blSdh6U20B0CULwa1WF0wpCX
+i/C=GB/O=w
ww.easysoft.com/OU=GT68879435/OU=See www.rapidssl.com/resources/cps (c
+)10/OU=Dom
ain Control Validated - RapidSSL(R)/CN=www.easysoft.com
Client-SSL-Cipher: DHE-RSA-AES256-SHA
Client-SSL-Socket-Class: IO::Socket::SSL
Client-Transfer-Encoding: chunked
Content-Style-Type: text/css
Link: </style/style.css>; rel="stylesheet"; type="text/css"
Link: </favicon.ico>; rel="icon"; type="image/x-icon"
Title: ODBC, JDBC and XML Driver Downloads for Windows, Unix, Linux an
+d Mac OS X
X-Meta-Description: Easysoft ODBC, JDBC and XML drivers let you access
+ Oracle, S
QL Server, Access, InterBase, Sybase, Firebird, RMS, ISAM, Coda and Li
+nc from Wi
ndows, Unix, Linux, Mac OS X and OpenVMS.
X-Meta-Keywords: easysoft,odbc,jdbc,xml,drivers,gateway,bridge,data,da
+tabases,fi
les,oracle,interbase,sybase,firebird,microsoft,sql server,access,coda,
+openvms,rm
s,isam,linc
X-Meta-Verify-V1: A80XwI68Uh8cIAFxkNSpouY1b8iRx/PeYQcfrBUotks=

Use of uninitialized value in split at /home/martin/perl5/perlbrew/per
+ls/perl-5.
14.0/lib/site_perl/5.14.0/Devel/Modlist.pm line 49.
AutoLoader             5.71
Carp                   1.20
Compress::Raw::Zlib   2.033
Config
Cwd                    3.36
Encode                 2.42
Encode::Alias          2.13
Encode::Config         2.05
Encode::Encoding       2.05
Encode::Locale         1.02
Errno                  1.13
Exporter             5.64_03
Exporter::Heavy      5.64_03
Fcntl                  1.11
File::Basename         2.82
File::Glob             1.12
File::GlobMapper      1.000
File::Spec             3.33
File::Spec::Unix       3.33
Getopt::Long           2.38
HTML::Entities         3.68
HTML::HeadParser       3.66
HTML::Parser           3.68
HTTP::Config           6.00
HTTP::Date             6.00
HTTP::Headers          6.00
HTTP::Message          6.02
HTTP::Request          6.00
HTTP::Response         6.01
HTTP::Status           6.00
I18N::Langinfo         0.08
IO                   1.25_04
IO::Compress::Base::Common  2.033
IO::Compress::Gzip::Constants  2.033
IO::Compress::Zlib::Extra  2.033
IO::File               1.15
IO::Handle             1.31
IO::Seekable            1.1
IO::Socket             1.32
IO::Socket::INET       1.31
IO::Socket::SSL        1.44
IO::Socket::UNIX       1.23
IO::Uncompress::Adapter::Inflate  2.033
IO::Uncompress::Base  2.033
IO::Uncompress::Gunzip  2.033
IO::Uncompress::RawInflate  2.033
LWP                    6.02
LWP::MemberMixin
LWP::Protocol          6.00
LWP::Protocol::http
LWP::Protocol::https   6.02
LWP::UserAgent         6.02
List::Util             1.23
Mozilla::CA          20110409
Net::HTTP              6.01
Net::HTTP::Methods     6.00
Net::HTTPS             6.00
Net::SSLeay            1.36
Scalar::Util           1.23
SelectSaver            1.02
Socket                 1.94
Storable               2.27
Symbol                 1.07
Time::Local          1.2000
URI                    1.58
URI::Escape            3.30
URI::Heuristic         4.19
URI::_generic
URI::_idna
URI::_punycode         0.03
URI::_query
URI::_server
URI::http
URI::https
XSLoader               0.13
base                   2.16
bytes                  1.04
constant               1.21
feature                1.20
integer                1.00
overload               1.13
unicore::Heavy.pl
unicore::To::Lower.pl
unicore::lib::Nt::De.pl
unicore::lib::Perl::SpacePer.pl
unicore::lib::Perl::Word.pl
utf8                   1.09
utf8_heavy.pl
vars                   1.02
warnings               1.12
warnings::register     1.02

      [download]
[reply]
[d/l]
[select]
        works for me i.e., it downloads the file but my starting example still does not work.

        Whoa, that doesn't make sense.

        lwp-download is essentially identical to your program (except the -d option says don't print the content).

        I have OpenSSL 1.0.0d

        Both work for me :/

[reply]
Re: "certificate verify failed" difference between Perl 5.14 and 5.10
by ikegami (Patriarch) on Jun 16, 2011 at 16:37 UTC

    Works for me.

    LWP 6.02, 5.14.0, ActiveState, x86, Windows

    LWP 6.02, 5.12.1, perlbrew, x86_64, Linux, OpenSSL-0.9.8g

    The certificate is by Equifax, and Equifax is in the .pem.

    Check that Mozilla::CA::SSL_ca_file() returns the right file and that you have permission to read the file.

[reply]
[d/l]

Back to Seekers of Perl Wisdom