alain_desilets has asked for the wisdom of the Perl Monks concerning the following question:

Here is a piece of code that tries to download a url using LWP:

use strict;

use warnings;
use LWP::UserAgent;
use IO::Socket::SSL;

my $url = "https://www.gov.nu.ca/";

do {
    my $user_agent = LWP::UserAgent->new(ssl_opts => { verify_hostname
+ => 0, SSL_verify_mode => IO::Socket::SSL::SSL_VERIFY_NONE, } );
    
    $user_agent->agent('Mozilla/5.0');

    my $request  = HTTP::Request->new( GET => $url );
    my $response = $user_agent->request($request);
    
    print "\n\ncontent:\n".$response->content;
    print "\n\ncode: ".$response->code."\n\nmessage: ".$response->mess
+age;
    
}

[download]

When I run it through my employer's VPN, it produces the following output: 

content:
... etc...
<TITLE>Untrusted SSL Server Certificate</TITLE>
... etc...

code: 503

message: Service Unavailable

[download]

Which is strange, given that I disabled the SSL certificate checking in my ssl_opts. Yet, if I curl the same URL, again through the VPN, I get the page's content.

Even stranger, if I log out of my employer's VPN, then the above script does work and produce the content of the page. So it seems that the VPN is somehow overriding my ssl_opts, but then, why is it not doing the same with curl?

Any idea about what is going on?

Thx.

Replies are listed 'Best First'.
Re: Cannot disable SSL certificate checking when using VPN
by Corion (Patriarch) on Jan 08, 2019 at 13:39 UTC 
    <TITLE>Untrusted SSL Server Certificate</TITLE>

    [download]

    LWP::UserAgent does not return HTML content in its error cases, so your (employers) VPN creates that HTML.

    Maybe if your code behaves more like Curl, it will be let through? Have you tried to set the appropriate HTTP headers? Especially setting the User-Agent header to something like curl/7.55.1 might already be enough.

    Compare what curl sends with what LWP::UserAgent sends, maybe through LWP::ConsoleLogger.

[reply]
[d/l]
[select]

      Thx Corion. I suspected that the HTML response (the one containing an error message) was generated by my employe's VPN, but I wasn't 100% sure.

      TThe script runs fine today, and I am not sure exactly what fixed it. Here is what I did:

      - Upgraded OSX from El Capitan to Mojave
      - Reinstalled Perl and all my apps required libraries using perlbrew

      I suspect that I was using a very old version of the SSL support libraries and that it wasn't compatible with my employee's VPN. By upgrading OSX and redoing the Perl instllation, I probably ended up installing a more recent version.

      It could also be that my employer changed something in its VPN, which is also plausible because this is the start of the year, and they may need to upgrade a bunch of things. But I lean more towards the first explanation (me upgrading the SSL libraries).

[reply]

Back to Seekers of Perl Wisdom