I have kept at it since I last posted data. Here is what I see in the log that is produced:

DEBUG: .../IO/Socket/SSL.pm:2503: new ctx 54371264
DEBUG: .../IO/Socket/SSL.pm:526: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:528: socket connected
DEBUG: .../IO/Socket/SSL.pm:550: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:586: not using SNI because hostname is unk
+nown
DEBUG: .../IO/Socket/SSL.pm:634: set socket to non-blocking to enforce
+ timeout=180
DEBUG: .../IO/Socket/SSL.pm:647: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:657: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:667: waiting for fd to become ready: SSL w
+ants a read first
DEBUG: .../IO/Socket/SSL.pm:687: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:647: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:657: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:667: waiting for fd to become ready: SSL w
+ants a read first
DEBUG: .../IO/Socket/SSL.pm:687: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:647: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:657: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:667: waiting for fd to become ready: SSL w
+ants a read first
DEBUG: .../IO/Socket/SSL.pm:687: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:2384: ok=0 cert=54354928
DEBUG: .../IO/Socket/SSL.pm:647: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:1757: SSL connect attempt failed

DEBUG: .../IO/Socket/SSL.pm:653: fatal SSL error: SSL connect attempt 
+failed error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certif
+icate verify failed
DEBUG: .../IO/Socket/SSL.pm:2537: free ctx 54371264 open=54371264
DEBUG: .../IO/Socket/SSL.pm:2542: free ctx 54371264 callback
DEBUG: .../IO/Socket/SSL.pm:2549: OK free ctx 54371264
2014/08/04 16:40:45> [http client] communication error: 500 Can't conn
+ect to 195.160.170.115:8443 (certificate verify failed)
500 Can't connect to 195.160.170.115:8443 (certificate verify failed)
[download]

I see two disconcerting things. First, I see "not using SNI because hostname is unknown". I do not know if this is a fatal error, or if there is anything I can do about it. Second, and what is the main, the fatal error, I see: "fatal SSL error: SSL connect attempt failed error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed" Would I be right in assuming that the three lines "waiting for fd to become ready: SSL wants a read first" represent three attempts to get the server's certificate? Would I be right in thinking that this is due to the sever not sending it's certificate? If so, what reasons may be behind that certificate not being sent? Is this exchange supposed to happen before or after the client sends the client certificate? I ask because I do not see anything that obviously related to the client sending it's own certificate.

I have also succeeded in getting the server's certificate. Here is most of what openssl tells me about it:

ted@linux-jp04:~/Work/Projects/FirstData> openssl s_client -connect 19
+5.160.170.115:8443 -showcerts
CONNECTED(00000003)
depth=1 C = LV, ST = Latvia, L = Riga, O = xxxxxxxxxxxx CN = ECOMM-tes
+t, emailAddress = xxxxxxxxxxx
verify error:num=19:self signed certificate in certificate chain
verify return:0
139748861212304:error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 aler
+t bad certificate:s3_pkt.c:1275:SSL alert number 42
139748861212304:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake 
+failure:s23_lib.c:177:
---
Certificate chain
 0 s:/C=LV/O=FDL/CN=lv-rtps-proxy-test.ne.1dc.com
   i:/C=LV/ST=Latvia/L=Riga/O=xxxxxxxxxx/CN=ECOMM-test/emailAddress=xx
+xxxxxxxxxxxxx
-----BEGIN CERTIFICATE-----
DELETED
-----END CERTIFICATE-----
 1 s:/C=LV/ST=Latvia/L=Riga/O=xxxxxxxxxxxxxx/CN=ECOMM-test/emailAddres
+s=xxxxxxxxxxxxx
   i:/C=LV/ST=Latvia/L=Riga/O=xxxxxxxxxxxxxxxxxxx/CN=ECOMM-test/emailA
+ddress=xxxxxxxxxxxxxxxxxx
-----BEGIN CERTIFICATE-----
DELETED
-----END CERTIFICATE-----
---
Server certificate
subject=/C=LV/O=FDL/CN=lv-rtps-proxy-test.ne.1dc.com
issuer=/C=LV/ST=Latvia/L=Riga/O=xxxxxxxxxxxxxxx/CN=ECOMM-test/emailAdd
+ress=xxxxxxxxxxxxxxx
---
Acceptable client certificate CA names
/C=LV/ST=Latvia/L=Riga/O=xxxxxxxxxxxxx/CN=ECOMM-test/emailAddress=xxxx
+xxxxxxxxxxxxxxx
/C=LV/O=FDL/CN=lv-rtps-proxy-test.ne.1dc.com
---
SSL handshake has read 3646 bytes and written 170 bytes
---
New, TLSv1/SSLv3, Cipher is EDH-RSA-DES-CBC3-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : EDH-RSA-DES-CBC3-SHA
    Session-ID: 53DFFBFBC58875853342EDD1EE387FCE42F0D98DE61F3C24D37FC5
+792C5AC3C2
    Session-ID-ctx: 
    Master-Key: DELETED
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1407187961
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate cha
+in)
---
ted@linux-jp04:~/Work/Projects/FirstData>
[download]

Of course I masked out the site operator's name and email address for obvious reasons

Alas, I just noticed a couple errors right at the beginning of the output, but I do not yet know what they mean. I do not think the presence of a self signed certificate in the mix is an issue as the operators of this site established their own CA, and use that to sign both their own server certificates and the client side certificates that their clients are to use when connecting, and, as I said, connection is possible ONLY through a VPN to their site. But, is this going to result in a verification failure (even though I have the root CA certificates for THEIR CA)?

Does this help?

Thanks

Ted

How do I get that version number? I tried running cpan, as root, and in it's help, it does not seem to give an option for viewing the version number of those packages that are installed. I them executed the command to upgrade IO::Socket::SSL, and cpan told me that all packages for it are up to date, if that helps. I would expect that result to depend on what platform I am using (OpenSuse 13.1, if that matters).

I can't get the server certificate, even with openssl s_client (it times out), though that happily gets the server certificate from every other server I need to communicate with using https. The operators of that site have sent me their root CA certificates (in two files, so I have to figure out how to tell LWP's useragent to use both of them, or so they say -- they went so far as to suggest I put them both in Java's keystore, as if that helps when writing Perl).

Here is my package, which is intended to encapsulate the complexities of using LWP with HTTPS:

package REJBlibUA::client;

use strict;
use warnings;

#use Net::SSL ();
use English qw(-no_match_vars);
use HTTP::Status;
use LWP::UserAgent;
use LWP::Protocol::https;
use HTTP::Request;
use HTTP::Request::Common;
use HTTP::Response;
use Log::Log4perl qw(:easy get_logger);
use UNIVERSAL::require;
use Encode qw(decode encode);

my $log_prefix = "[http client] ";

sub new {
    my ($class, %params) = @_;

    die "non-existing certificate file $params{ca_cert_file}"
        if $params{ca_cert_file} && ! -f $params{ca_cert_file};

    die "non-existing certificate directory $params{ca_cert_dir}"
        if $params{ca_cert_dir} && ! -d $params{ca_cert_dir};

    my $self = {
        logger       => '',
        user         => $params{user},
        password     => $params{password},
        timeout      => $params{timeout} || 180,
        ssl_set      => 0,
        no_ssl_check => $params{no_ssl_check},
        ca_cert_dir  => $params{ca_cert_dir},
        ca_cert_file => $params{ca_cert_file},
    SSL_cert_file => $params{SSL_cert_file},
    SSL_key_file => $params{SSL_key_file},
    };
    bless $self, $class;

    my $conf = q(
        log4perl.logger                    = INFO, FileApp, ScreenApp
#        log4perl.logger                    = TRACE, FileApp, ScreenAp
+p
        log4perl.appender.FileApp          = Log::Log4perl::Appender::
+File
        log4perl.appender.FileApp.filename = lwp.log
        log4perl.appender.FileApp.layout   = PatternLayout
        log4perl.appender.FileApp.layout.ConversionPattern = %d> %m%n
        log4perl.appender.ScreenApp          = Log::Log4perl::Appender
+::Screen
        log4perl.appender.ScreenApp.stderr   = 0
        log4perl.appender.ScreenApp.layout   = PatternLayout
        log4perl.appender.ScreenApp.layout.ConversionPattern = %d> %m%
+n
    );

# Initialize logging behaviour
    Log::Log4perl->init( \$conf );
#    Log::Log4perl->infiltrate_lwp();
    *trace = *INFO;
    *conns = *DEBUG;
    *debug = *DEBUG;
    $self->{'logger'} = get_logger();
    # create user agent
    $self->{ua} = LWP::UserAgent->new(
        parse_head => 0, # No need to parse HTML
        keep_alive => 1,
        requests_redirectable => ['POST', 'GET', 'HEAD']
    );
    $self->{ua}->ssl_opts(verify_hostname => 1, SSL_verify_mode => 1);
    if ($params{proxy}) {
        $self->{ua}->proxy(['http', 'https'], $params{proxy});
    }  else {
        $self->{ua}->env_proxy();
    }

    $self->{ua}->timeout($self->{timeout});

    return $self;
}

sub request { 
   my ($self, $request, $file) = @_;
    # $request is a HTTP::Request object, created with only the URL
    # $file is a message, normally an XML file
    my $logger  = $self->{logger};

    my $url = $request->uri();
    my $scheme = $url->scheme();
   print "\$url = $url\n\t\$scheme = $scheme\n";
   print "\t\$self->{ssl_set} = ",$self->{ssl_set},"\n";
   print "\t\$self->{ca_cert_dir} = ",$self->{ca_cert_dir},"\n";
   print "\t\$self->{ca_cert_file} = ",$self->{ca_cert_file},"\n";
    $self->_setSSLOptions() if $scheme eq 'https' && !$self->{ssl_set}
+;

    my $result = HTTP::Response->new( 500 );
    eval {
       $result = $self->{ua}->request($request, $file);
    };
    # check result first
    if (!$result->is_success()) {
        # authentication required
        if ($result->code() == 401) {
            if ($self->{user} && $self->{password}) {
                $logger->debug(
                    $log_prefix .
                    "authentication required, submitting credentials"
                );
                # compute authentication parameters
                my $header = $result->header('www-authenticate');
                my ($realm) = $header =~ /^Basic realm="(.*)"/;
                my $host = $url->host();
                my $port = $url->port() ||
                   ($scheme eq 'https' ? 443 : 80);
                $self->{ua}->credentials(
                    "$host:$port",
                    $realm,
                    $self->{user},
                    $self->{password}
                );
                # replay request
                eval {
                    if ($OSNAME eq 'MSWin32' && $scheme eq 'https') {
                        alarm $self->{timeout};
                    }
                    $result = $self->{ua}->request($request, $file);
                };
                if (!$result->is_success()) {
                    $logger->error(
                        $log_prefix .
                        "authentication required, wrong credentials"
                    );
                }
            } else {
                # abort
                $logger->error(
                    $log_prefix .
                    "authentication required, no credentials available
+"
                );
            }
        } else {
            $logger->error(
                $log_prefix .
                "communication error: " . $result->status_line()
            );
        }
    }

    return $result;
}

sub _setSSLOptions {
    my ($self) = @_;

    # SSL handling
    $ENV{HTTPS_DEBUG} = 1; 
    if ($self->{no_ssl_check}) {
       # LWP 6 default behaviour is to check hostname
       # Fedora also backported this behaviour change in its LWP5 pack
+age, so
       # just checking on LWP version is not enough
       $self->{ua}->ssl_opts(verify_hostname => 0, SSL_verify_mode => 
+0)
           if $self->{ua}->can('ssl_opts');
    } else {
        # only IO::Socket::SSL can perform full server certificate val
+idation,
        # Net::SSL is only able to check certification authority, and 
+not
        # certificate hostname
#        IO::Socket::SSL->require();
#        IO::Socket::SSL->require('debug3');
    use IO::Socket::SSL qw(debug3); 
        die
            "failed to load IO::Socket::SSL, "                 .
            "unable to perform SSL certificate validation.\n"  .
            "You can use 'no-ssl-check' option to disable it."
            if $EVAL_ERROR;

#        if ($self->{logger}{debug} >= 3) {
#            $Net::SSLeay::trace = 2;
#        }
    print "\t\t\$LWP::VERSION = $LWP::VERSION\n";

        if ($LWP::VERSION >= 6) {
        print "\t\tSetting cert dir and file if available\n";
            $self->{ua}->ssl_opts(SSL_ca_file => $self->{ca_cert_file}
+)
                if $self->{ca_cert_file};
            $self->{ua}->ssl_opts(SSL_ca_path => $self->{ca_cert_dir})
                if $self->{ca_cert_dir};
            $self->{ua}->ssl_opts(SSL_cert_file => $self->{SSL_cert_fi
+le})
                if $self->{SSL_cert_file};
            $self->{ua}->ssl_opts(SSL_key_file => $self->{SSL_key_file
+})
                if $self->{SSL_key_file};
        }
    }

    $self->{ssl_set} = 1;
}

1;
[download]

And here is the calling code, used to test it:

#!/usr/bin/perl

use HTTP::Request;
use HTTP::Response;

use lib './Work';

use REJBlibUA::client;

my $method = "POST";
my @requests = (HTTP::Request->new( $method,'https://www.google.ca'), 
                HTTP::Request->new( $method,'https://gremlin.site/cgi-
+bin/printenv.pl'),
        HTTP::Request->new( $method,'https://byerspublishing.com'),
        HTTP::Request->new( $method,'https://195.160.170.115:8443/soap
+proxy/PaymentServer'));

my $cnt = 0;
foreach my $r (@requests) {
    my %rp;
    if ($cnt == 1) {
    $rp{'ca_cert_file'} = 'rootCA.pem';  
    $rp{'ca_cert_dir'} = '.';
    $rp{'SSL_cert_file'} = 'client.crt';
    $rp{'SSL_key_file'} = 'client.key';
    }
    if ($cnt == 3) {
    #next;
    $rp{'ca_cert_file'} = 'ECOMM_old_test.pem';  
    $rp{'ca_cert_dir'} = '.';
    $rp{'SSL_cert_file'} = 'OnsoftArch_test.pem';
    $rp{'SSL_key_file'} = 'onsoftarch_test.key';
    }

    my $c = REJBlibUA::client->new(%rp);
    
    my $resp = $c->request($r);

    if ($resp->is_success) {
    print $resp->decoded_content;
    } else {
    print STDERR $resp->status_line, "\n";
    }
    $cnt += 1;
}
[download]

https://gremlin.site/cgi-bin/printenv.pl works perfectly. https://byerspublishing.com fails because it uses only the default certificate Apache comes with (until I buy a proper certificate for it), and thus the server certificate is not trusted. https://195.160.170.115:8443/soapproxy/PaymentServer is the real problem, but it is accessable only through a VPN to the operator's site. I can ping that server, but openssl times out when I try to get the server's certificate. I suspect their server is not sending it's certificate, for whatever reason. But then, I don't know the sequence of events that must happen when the handshaking is to involve both client and server side certificate validation. Is the client supposed to send it's certificate first, and is it supposed to send the certificate for the CA that signed it (even if the server already has that CA certificate), before the server sends it's certificate, or after?

I suppose the fact the problem server is accessable only through a VPN makes diagnosing the problem, and that I can't seem to get it's certificate, makes diagnosing the problem more challenging.

How do I use $IO::Socket::SSL::DEBUG?

Thanks

Ted

> How do I get that version number?
This works the same as with other perl modules, just print $IO::Socket::SSL::VERSION. But according to your description you must have the newest version, that is 1.997.

> I can't get the server certificate, even with openssl s_client (it times out), though that happily gets the server certificate from every other server I need to communicate with using https.
If s_client cannot connect then there is probably no valid SSL server at the other end. But, it might also be a buggy server or a server with a bad SSL accelerator in front. Try to restrict the protocol in s_client with the -ssl3 option and see if this helps.

Thanks

I finally did solve the problem with connecting to that server using openssl s_client. I can now reliably connect using that

The problem that remains is that although I provide the same information to my client, written in Perl, the server fails to send it's certificate. This is a puzzle since I have no problem using the same code when connecting to the servers I control, which are set up to require client side certificates also. In those cases, both certificates are exchanged and verified successfully, and the requested data retrieved. Is there any chance that openssl s_client does anything that the LWP user agent, using IO::SOCKET::SSL doesn't do?

Thanks

Ted