ffrost has asked for the wisdom of the Perl Monks concerning the following question:

Objective: To connect from my machine to an external server using HTTPS. The connection will use the GET verb and the external server should return a message back to the client. As a security mechanism, the certificate for my client must be sent to the server as authentication.

I'm using LWP to get the response from the external server. I've run into two problems I'm not sure of. First, I am unsure if the LWP program is actually loading the certificate, which is in DER format. Second, since the server is responding with a 403 error I don't think they are getting my certificate. Am I loading the certificate correctly to send from my Windows machine to their external IIS server?

The DER certificate is valid and has not expired. It is not a self-signed certificate.

use LWP::UserAgent;
use Data::Dumper;
use Cwd;

$cdir = getcwd;
$endpoint = ‘https://omit/commotest';

$ua = LWP::UserAgent->new;
$ua->ssl_opts(SSL_cert_file => "$cdir\\der.cer");
$ua->ssl_opts(SSL_use_cert => '1');

$response = $ua->get($endpoint);           

if ($response->is_success) 
 {  print Dumper $response; }
else { print "Error: " . $response->status_line, "\n"; }

[download]

ICYMI: Here's the solution

$ua->ssl_opts(SSL_cert_file => 'privatekey.pfx');
$ua->ssl_opts(SSL_passwd_cb => sub { return "passwordvaluehere"; } );
$ua->ssl_opts(SSL_use_cert => '1');

[download]

Replies are listed 'Best First'.
Re: Issue with LWP loading client certificate
by haj (Vicar) on Jun 17, 2021 at 21:43 UTC

    DER format is fine for certificates. But then, many things can go wrong with SSL. To find out what happens to your certificate switch on SSL debugging by setting $IO::Socket::SSL::DEBUG=2 in your code - see IO::Socket::SSL. It is a bit verbose, but should clearly indicate whether your client certificate has been used for the request.

[reply]

      Tried this one and it's not verbose enough to verify that the certificate is loaded/sent. This was at the debug set to 4. 

      DEBUG: .../IO/Socket/SSL.pm:2649: new ctx 47286544
DEBUG: .../IO/Socket/SSL.pm:562: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:564: socket connected
DEBUG: .../IO/Socket/SSL.pm:586: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:619: using SNI with hostname OMIT
DEBUG: .../IO/Socket/SSL.pm:654: request OCSP stapling
DEBUG: .../IO/Socket/SSL.pm:673: set socket to non-blocking to enforce
+ timeout=180
DEBUG: .../IO/Socket/SSL.pm:686: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:689: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:699: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:709: waiting for fd to become ready: SSL w
+ants a read first
DEBUG: .../IO/Socket/SSL.pm:729: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:686: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:2552: did not get stapled OCSP response
DEBUG: .../IO/Socket/SSL.pm:2505: ok=1 cert=52520624
DEBUG: .../IO/Socket/SSL.pm:2505: ok=1 cert=52517456
DEBUG: .../IO/Socket/SSL.pm:2505: ok=1 cert=53422432
DEBUG: .../IO/Socket/SSL.pm:1594: scheme=www cert=53422432
DEBUG: .../IO/Socket/SSL.pm:1604: identity=OMIT cn=OMIT alt=2 OMIT
DEBUG: .../IO/Socket/SSL.pm:689: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:699: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:709: waiting for fd to become ready: SSL w
+ants a read first
DEBUG: .../IO/Socket/SSL.pm:729: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:686: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:689: done Net::SSLeay::connect -> 1
DEBUG: .../IO/Socket/SSL.pm:744: ssl handshake done
DEBUG: .../IO/Socket/SSL.pm:2552: did not get stapled OCSP response
DEBUG: .../IO/Socket/SSL.pm:2505: ok=1 cert=52520624
DEBUG: .../IO/Socket/SSL.pm:2505: ok=1 cert=52517456
DEBUG: .../IO/Socket/SSL.pm:2505: ok=1 cert=53426480
DEBUG: .../IO/Socket/SSL.pm:1594: scheme=www cert=53426480
DEBUG: .../IO/Socket/SSL.pm:1604: identity=OMIT cn=OMIT alt=2 OMIT
DEBUG: .../IO/Socket/SSL.pm:2682: free ctx 47286544 open=47286544
DEBUG: .../IO/Socket/SSL.pm:2687: free ctx 47286544 callback
DEBUG: .../IO/Socket/SSL.pm:2694: OK free ctx 47286544
Error: 403 Forbidden

      [download]
[reply]
[d/l]

        Sorry - I thought there would be more output. Probably I misremembered from tests using OpenSSL on the command line, which is a bit cumbersome with Windows (but works fine with Cygwin).

        So there are two more suggestions:

        • Fire your certificate against an online test service, for example https://certauth.idrix.fr/, by just changing the endpoint URL in your code. This service requires client certificates and responds by providing a technical description of the certificate you provided.
        • Import the client certificate and the corresponding CA certificate into a browser and then read the page.

        Maybe one of those gives more insight?

[reply]
Re: Issue with LWP loading client certificate
by bliako (Abbot) on Jun 18, 2021 at 06:33 UTC

    adding to haj's debugging suggestion, this can make LWP more verbose:

    use LWP::ConsoleLogger::Easy qw( debug_ua );
# $ua is your LWP, 5 is verbosity
debug_ua($ua, 5);

    [download]

    bw, bliako

[reply]
[d/l]
Re: Issue with LWP loading client certificate
by jo37 (Curate) on Jun 18, 2021 at 13:38 UTC

    For authentication you need a keyfile in addition to a certfile.

    Greetings,
    -jo

    $gryYup$d0ylprbpriprrYpkJl2xyl~rzg??P~5lp2hyl0p$
[reply]

      That's true for DER files, but not for all certificate formats (PEM and PKCS#12 can have both key and certificate rolled into one file). But if that was the problem, then IO::Socket::SSL would complain loudly and the handshake wouldn't complete. So I rather guess that the server didn't request a client certificate.

[reply]

      A keyfile? Something like ... 

      $ua->ssl_opts(SSL_key_file => 'privatekey.pem');

      [download]

      So, would this load the cert/keyfile then?

      $ua->ssl_opts(SSL_key_file => 'privatekey.pem');
$ua->ssl_opts(SSL_cert_file => 'cert.pem');

      [download]

      And, they have to be .pem formatted files?

[reply]
[d/l]
[select]
        And, they have to be .pem formatted files?

        Not necessarily. The documentation says:

        Supported file formats are PEM, DER and PKCS#12, where PEM and PKCS#12 can contain the certificate and the chain to use, while DER can only contain a single certificate.

        🦛

[reply]

        Yes, this should work with a privatekey in PEM format. It needs to be unencrypted, though.

        UPDATE:
        You may as well put the cert and the privkey in PEM format into the certfile or you may use a PKCS#12 as certfile. And you may give a callback sub to provide the passphrase for the private key via SSL_password_cb. See IO::Socket::SSL.

        Greetings,
        -jo

        $gryYup$d0ylprbpriprrYpkJl2xyl~rzg??P~5lp2hyl0p$
[reply]
[d/l]

      Further down the rabbit hole we go. Open one door and another presents itself.

      We cannot have unencrypted .pfx files for the private key on the server. But, there's notation stating the password for the private key can get passed into the IO::Socket with a subroutine. This part doesn't make sense to me. From what I've found this is how it's put together, but it's still failing.

      use LWP::UserAgent;
use Data::Dumper;
use IO::Socket::SSL qw(debug4);

$endpoint = ‘https://omit/commotest';

$ua = LWP::UserAgent->new;
$ua->ssl_opts(SSL_cert_file => 'der.cer');
$ua->ssl_opts(SSL_key_file => 'privatekey.pfx');
$ua->ssl_opts(SSL_passwd_cb => sub { return "passwordvaluehere"; } );
$ua->ssl_opts(SSL_use_cert => '1');

$response = $ua->get($endpoint);           

if ($response->is_success) 
 {  print Dumper $response; }
else { print "Error: " . $response->status_line, "\n"; }

      [download]

      But now I'm getting the following response ... We're using a simple passphrase for now to test with, no symbols. Research online showed other users had issues with complex passphrases and the decryption.

      DEBUG: .../IO/Socket/SSL.pm:2411: Failed to load key from file (no PEM
+ or DER)
SSL error:  4824: 1 - error:0606F076:digital envelope routines:EVP_PKC
+S82PKEY:unsupported private key algorithm
SSL error:  4824: 2 - error:140B000D:SSL routines:SSL_CTX_use_PrivateK
+ey_file:ASN1 lib

      [download]
[reply]
[d/l]
[select]

        I would guess that your privatekey.pfx is all you need. That extension is used for PKCS#12 files, which are containers usually having the certificate and the private key embedded.

        An extra SSL_key_file must be PEM or DER (that's what the message says) but you should make progress by providing the .pfx file as SSL_cert_file and drop the SSL_key_file:

        $ua = LWP::UserAgent->new;
$ua->ssl_opts(SSL_cert_file => 'privatekey.pfx');
$ua->ssl_opts(SSL_passwd_cb => sub { return "passwordvaluehere"; } );
$ua->ssl_opts(SSL_use_cert => '1');

        [download]
[reply]
[d/l]
Re: Issue with LWP loading client certificate
by sectokia (Friar) on Jun 21, 2021 at 12:34 UTC
    If you are getting 403, this means SSL was established. Your problem is probably with your request. What happens if you add your client cert to a web browser (typically as a client profile), and make the request?
[reply]

Back to Seekers of Perl Wisdom