HTTPS connection with LWP and self-signed certificate

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I have a small perl scripts which retrieves data from a a webserver running in my own network. The server uses https with a self signed certificate. I'm currently using LWP::UserAgent with "ssl_opts('verify_hostname' => 0) which works but is insecure. Since I'm only making connections to this single server I am able to hardcode information about its certificate in the script. From what I've read I should be able to supply the certificate using the "SSL_ca_file" option. I'm using an export of the certificate made by firefox which I stored in the same directory as the script as "cert.pem". The code is given below:

#!/usr/bin/perl
use strict;
use warnings;

use LWP::UserAgent;
use Data::Dumper;
use IO::Socket::SSL qw(debug3);

my $server="192.168.100.222";
my $port="443";

my $cert = "cert.pem"; #using relative or absolute paths doesn't make 
+a difference
my $ua  = LWP::UserAgent->new();
$ua->ssl_opts('SSL_ca_file' => $cert); #doesn't work
#$ua->ssl_opts('verify_hostname' => 0); #works

my $response= $ua->get("https://$server:$port/");
print $response->as_string;
[download]

The connection fails with the following debug output (stripped certificate identity)

DEBUG: .../IO/Socket/SSL.pm:1653: new ctx 39194656
DEBUG: .../IO/Socket/SSL.pm:363: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:365: socket connected
DEBUG: .../IO/Socket/SSL.pm:383: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:433: set socket to non-blocking to enforce
+ timeout=180
DEBUG: .../IO/Socket/SSL.pm:446: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:456: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:466: waiting for fd to become ready: SSL w
+ants a read first
DEBUG: .../IO/Socket/SSL.pm:486: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:1641: ok=1 cert=38910576
DEBUG: .../IO/Socket/SSL.pm:1201: scheme=www cert=38910576
DEBUG: .../IO/Socket/SSL.pm:1210: identity=192.168.100.222 cn=________
+____ alt=
DEBUG: .../IO/Socket/SSL.pm:446: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:1328: SSL connect attempt failed with unkn
+own error error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:cer
+tificate verify failed

DEBUG: .../IO/Socket/SSL.pm:452: fatal SSL error: SSL connect attempt 
+failed with unknown error error:14090086:SSL routines:SSL3_GET_SERVER
+_CERTIFICATE:certificate verify failed
DEBUG: .../IO/Socket/SSL.pm:1328: IO::Socket::IP configuration failed 
+error:00000000:lib(0):func(0):reason(0)

DEBUG: .../IO/Socket/SSL.pm:1690: free ctx 39194656 open=39194656
DEBUG: .../IO/Socket/SSL.pm:1695: free ctx 39194656 callback
DEBUG: .../IO/Socket/SSL.pm:1698: OK free ctx 39194656
500 Can't connect to 192.168.100.222:443
Content-Type: text/plain
Client-Date: Tue, 06 Jan 2015 14:53:10 GMT
Client-Warning: Internal response

Can't connect to 192.168.100.222:443
[download]

If someone could tell me what I'm doing wrong or if there is a different way to securely connect to a server using a self-signed certificate it would be highly appreciated! OS: Debian Wheezy Perl 5.14.2

Comment on HTTPS connection with LWP and self-signed certificate Select or Download Code

Replies are listed 'Best First'.
Re: HTTPS connection with LWP and self-signed certificate by noxxi (Pilgrim) on Jan 07, 2015 at 16:59 UTC
I don't know what you did and how your certificate looks like, but this is what I did and which works with the current version of IO::Socket::SSL. # create a self-signed certificate perl -MIO::Socket::SSL::Utils -e ' my ($c,$k) = CERT_create(CA => 1, subject => { CN => q[foo.bar]}); print PEM_cert2string($c).PEM_key2string($k)' \ > self-signed.pem # start a server with this certificate (default port 4433) openssl s_server -cert self-signed.pem -key self-signed.pem & # connect to this server perl -Ilib -MIO::Socket::SSL=debug4 -e ' my $cl = IO::Socket::SSL->new( PeerAddr => q[127.0.0.1:4433], SSL_hostname => q[foo.bar], SSL_ca_file => q[self-signed.pem] ) or die "$SSL_ERROR,$!"; warn "SUCCESS cipher=".$cl->get_cipher();' [download] This gives me (among all the debug output) "SUCCESS cipher=ECDHE-RSA-AES128-SHA256" which means, that the SSL connection was successfully established. You might check with your certificate to make sure, that the problem is not the certificate itself.	[reply] [d/l]
Re^2: HTTPS connection with LWP and self-signed certificate by Anonymous Monk on Jan 07, 2015 at 23:10 UTC
I upgraded my IO::Socket::SSL to the latest version, it took a while since I also had to upgrade a bunch of other stuff to get it to work, but at least now I have all the utilities. I ran your code to generate a certificate, setup the server and connect to it, that all worked. Next I ran the server on the host which I'm trying to connect to and modified the ip for the client, still worked. Then I configured apache to use that exact certificate and and again it worked! Convinced that all my troubles were over I tried to execute my script from the original post to see if it would also work with the new certificate but... DEBUG: .../IO/Socket/SSL.pm:2555: new ctx 34454560 DEBUG: .../IO/Socket/SSL.pm:539: socket not yet connected DEBUG: .../IO/Socket/SSL.pm:541: socket connected DEBUG: .../IO/Socket/SSL.pm:563: ssl handshake not started DEBUG: .../IO/Socket/SSL.pm:599: not using SNI because hostname is unk +nown DEBUG: .../IO/Socket/SSL.pm:631: request OCSP stapling DEBUG: .../IO/Socket/SSL.pm:650: set socket to non-blocking to enforce + timeout=180 DEBUG: .../IO/Socket/SSL.pm:663: Net::SSLeay::connect -> -1 DEBUG: .../IO/Socket/SSL.pm:673: ssl handshake in progress DEBUG: .../IO/Socket/SSL.pm:683: waiting for fd to become ready: SSL w +ants a read first DEBUG: .../IO/Socket/SSL.pm:703: socket ready, retrying connect DEBUG: .../IO/Socket/SSL.pm:2458: did not get stapled OCSP response DEBUG: .../IO/Socket/SSL.pm:2411: ok=1 cert=34885312 DEBUG: .../IO/Socket/SSL.pm:1559: scheme=www cert=34885312 DEBUG: .../IO/Socket/SSL.pm:1569: identity=192.168.100.222 cn=________ +_____ alt= DEBUG: .../IO/Socket/SSL.pm:1769: hostname verification failed DEBUG: .../IO/Socket/SSL.pm:663: Net::SSLeay::connect -> -1 DEBUG: .../IO/Socket/SSL.pm:1780: SSL connect attempt failed DEBUG: .../IO/Socket/SSL.pm:1785: SSL connect attempt failed error:140 +90086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify fai +led DEBUG: .../IO/Socket/SSL.pm:669: fatal SSL error: SSL connect attempt +failed error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certif +icate verify failed DEBUG: .../IO/Socket/SSL.pm:1769: IO::Socket::IP configuration failed DEBUG: .../IO/Socket/SSL.pm:2588: free ctx 34454560 open=34454560 DEBUG: .../IO/Socket/SSL.pm:2593: free ctx 34454560 callback DEBUG: .../IO/Socket/SSL.pm:2600: OK free ctx 34454560 500 Can't connect to 192.168.100.222:4433 (certificate verify failed) Content-Type: text/plain Client-Date: Wed, 07 Jan 2015 23:00:51 GMT Client-Warning: Internal response Can't connect to 192.168.100.222:4433 (certificate verify failed) SSL connect attempt failed error:14090086:SSL routines:SSL3_GET_SERVER +_CERTIFICATE:certificate verify failed at /usr/local/share/perl/5.14. +2/LWP/Protocol/http.pm line 49 [download] On the serverside openssl s_server said: `Using default temp DH parameters Using default temp ECDH parameters ACCEPT bad gethostbyaddr ERROR 140707196729000:error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 aler +t certificate unknown:s3_pkt.c:1256:SSL alert number 46 shutting down SSL CONNECTION CLOSED ACCEPT` [download] trying to connect to apache with the new cert also failed. This leaves me to think that there is something wrong with (the way I'm using) LWP::UserAgent. Is it possible to handle the ssl connection with IO::Socket::SSL directly but still have all the LWP::UserAgent functionality for my interaction with the server after the connection is established?	[reply] [d/l] [select]
Re^3: HTTPS connection with LWP and self-signed certificate by noxxi (Pilgrim) on Jan 08, 2015 at 06:22 UTC
Is it possible to handle the ssl connection with IO::Socket::SSL directly but still have all the LWP::UserAgent functionality for my interaction with the server after the connection is established? No, LWP wants to have full control over the socket and does not allow to create a connection using an already established socket. Apart from that, the following code works for me with LWP::UserAgent 6.05 and LWP::Protocol::https 6.04 (with the modification done in Ubuntu 14.04): `use strict; use warnings; use LWP::UserAgent; my $ua = LWP::UserAgent->new; $ua->ssl_opts( SSL_ca_file => 'self-signed.pem'); # set verification name explicitly for this test because the # URL does not contain the correct name $ua->ssl_opts( SSL_verfifycn_name => 'foo.bar'); my $res = $ua->get('https://127.0.0.1:4433'); print $res->as_string;` [download] In this case s_server is started with -WWW, that is: `openssl s_server -cert self-signed.pem -key self-signed.pem -WWW` [download] Which leaves the question about the version of LWP::UserAgent and LWP::Protocol::https you are using.	[reply] [d/l] [select]
Re^4: HTTPS connection with LWP and self-signed certificate by Anonymous Monk on Jan 08, 2015 at 10:26 UTC
Re^5: HTTPS connection with LWP and self-signed certificate by Anonymous Monk on Jan 08, 2015 at 10:34 UTC
Re^4: HTTPS connection with LWP and self-signed certificate by Anonymous Monk on Jan 08, 2015 at 07:43 UTC
Re: HTTPS connection with LWP and self-signed certificate ( openssl/ssldump ) by Anonymous Monk on Jan 06, 2015 at 20:22 UTC
What are the details of your certificate, and how do they match the details of your server? Sounds like the hostname reported by the server doesn't match the certificate ... not sure how to check that I would also try to use the openssl/ssldump programs to try to figure out why certification failed ... see Debugging SSL communications	[reply]
Re^2: HTTPS connection with LWP and self-signed certificate ( openssl/ssldump ) by Anonymous Monk on Jan 07, 2015 at 11:44 UTC
There was indeed a mismatch in the hostnames. I changed the hostname of the server and restarted apache. This however did not effect the error nor the debug information shown. I tried connecting via openssl s_client to get some more information but I didn't really see anything strange. It mentioned that the cert was self-signed but didn't complain, typing "GET / HTTP/1.0" gave me the page just as going to it in a browser does. The relevant output of "openssl s_client -connect 192.168.100.222:443" is given below CONNECTED(00000003) depth=0 C = NL, ST = Some-State, O = _______, CN = _____________, emai +lAddress = _______________ verify error:num=18:self signed certificate verify return:1 depth=0 C = NL, ST = Some-State, O = _______, CN = _____________, emai +lAddress = _______________ verify return:1 --- Certificate chain 0 s:/C=NL/ST=Some-State/O=_______/CN=_____________/emailAddress=_____ +__________ i:/C=NL/ST=Some-State/O=_______/CN=_____________/emailAddress=_____ +__________ --- Server certificate -----BEGIN CERTIFICATE----- . .(Same cert as the cert.pem supplied to the perl code) . -----END CERTIFICATE----- subject=/C=NL/ST=Some-State/O=______/CN=_____________/emailAddress=___ +____________ issuer=/C=NL/ST=Some-State/O=______/CN=_____________/emailAddress=____ +___________ --- No client certificate CA names sent --- SSL handshake has read 1820 bytes and written 498 bytes --- New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1.2 Cipher : DHE-RSA-AES256-GCM-SHA384 Session-ID: 7E2DD78B639825C10C28C8F56AF56100E4CB67155BB1348EB7C32E +10F02C2066 Session-ID-ctx: Master-Key: 4D6A8EA327F98D1DF703D299CA29CEA5776A5FE7DB4FC32F4D5D0A +DEE58FCAB7D24560107E5ECF0DBE12AEE1A8900321 Key-Arg : None PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 300 (seconds) TLS session ticket: 0000 - a8 71 f5 e4 bd f7 89 bf-cf 9d 4c d8 38 7e 0c 76 .q....... +.L.8~.v 0010 - 54 02 44 c3 02 03 d0 3f-74 05 3f db 16 01 26 1f T.D....?t +.?...&. 0020 - 05 da 8e 34 d7 a5 20 a8-9d 81 69 6c 74 c7 eb 26 ...4.. .. +.ilt..& 0030 - 38 4d b9 fa 2f 59 8b 86-c0 cb b9 f2 72 26 e6 96 8M../Y... +...r&.. 0040 - 67 7c ca 19 6d 28 29 68-19 8b 3b d3 3d de e3 22 g\|..m()h. +.;.=.." 0050 - 10 88 0b 47 39 f5 20 96-4e a9 29 b2 78 97 a7 be ...G9. .N +.).x... 0060 - f9 d2 88 95 17 65 21 6e-f4 b5 80 ec 67 c4 ae af .....e!n. +...g... 0070 - c1 06 a8 03 21 54 28 5a-bb 9c 41 12 b3 81 27 73 ....!T(Z. +.A...'s 0080 - 59 86 3f ec 9d 9b 57 06-8d 59 bb 5e fc f2 4b 24 Y.?...W.. +Y.^..K$ 0090 - f7 46 37 64 82 8c 52 46-d1 ee 82 9b c7 c4 0b 12 .F7d..RF. +....... 00a0 - 35 cf 7e 89 3f ad cd 97-da d1 e2 ee 71 03 5c 50 5.~.?.... +...q.\P 00b0 - d2 60 59 1e ad f1 71 de-a4 7b 25 bf 45 0a 36 1a .`Y...q.. +{%.E.6. Start Time: 1420627933 Timeout : 300 (sec) Verify return code: 18 (self signed certificate) --- GET / HTTP/1.0 HTTP/1.1 200 OK Date: Wed, 07 Jan 2015 10:52:14 GMT Server: Apache/2.2.22 (Debian) Last-Modified: Tue, 09 Jul 2013 11:04:40 GMT ETag: "2069e-b1-4e11220a261a8" Accept-Ranges: bytes Content-Length: 177 Vary: Accept-Encoding Connection: close Content-Type: text/html <html><body><h1>It works!</h1> <p>This is the default web page for this server.</p> <p>The web server software is running but no content has been added, y +et.</p> </body></html> closed [download] Maybe I should explicitly tell openssl s_client to use the certificate so its behavior is more similar to the perl code, but I couldn't find out how to do so..	[reply] [d/l]
Re^3: HTTPS connection with LWP and self-signed certificate ( openssl/ssldump ) by Anonymous Monk on Jan 07, 2015 at 11:55 UTC
Hmm... https://metacpan.org/pod/IO::Socket::SSL#Common-Problems-with-SSL says Also, util/analyze-ssl.pl in the distribution might be a helpful tool when debugging SSL problems, as do the openssl command line tool and a check with a different SSL implementation (e.g. a web browser). So I'd try that next :)	[reply]
Re^4: HTTPS connection with LWP and self-signed certificate ( openssl/ssldump ) by Anonymous Monk on Jan 07, 2015 at 12:46 UTC
Re^5: HTTPS connection with LWP and self-signed certificate ( openssl/ssldump ) by Corion (Patriarch) on Jan 07, 2015 at 13:30 UTC
Some notes below your chosen depth have not been shown here
Re^5: HTTPS connection with LWP and self-signed certificate ( openssl/ssldump ) by Anonymous Monk on Jan 07, 2015 at 23:39 UTC
Re: HTTPS connection with LWP and self-signed certificate by locked_user sundialsvc4 (Abbot) on Jan 06, 2015 at 19:00 UTC
Look at the logs of the server you are attempting to connect to. It is probably getting the cert but rejecting it. SSL is designed to work exactly one way and to give minimal, cryptic diagnostic information. (After all, “you could be Eve.”)
Re^2: HTTPS connection with LWP and self-signed certificate by Anonymous Monk on Jan 07, 2015 at 10:57 UTC
I checked both the /var/log/apache2/access.log and error.log. There was no sign of the connection in either of them. I guess it didn't show up in the access log because the connection was never completed?	[reply]