YarNik has asked for the wisdom of the Perl Monks concerning the following question:

Hello! I'm trying to get a https page, half of the domains is OK. But with the other half I get the error:

500 Can not connect to domen: 443 (connect: Network is unreachable); Client-Warning: Internal response

The script is standard, I tried to change the agent and different versions of ssl_opts: 
        use LWP::UserAgent;
        $ua = LWP::UserAgent->new(
#            agent                 => 'Mozilla/5.0 (X11; U; Linux i686
+; en-US; rv:1.9.0.5) Gecko/2008120121 Firefox/3.0.5',
#            ssl_opts => { verify_hostname => 0, SSL_verify_mode => 0,
+ SSL_verifycn_scheme => 'none' },
        );
        $response = $ua->head("$get_url");

        print $response->dump();

[download]
P.s. domain example microsoft.com, google.com ( bing - it is OK. )

Replies are listed 'Best First'.
Re: Error 500 in LWP
by hippo (Archbishop) on Aug 17, 2017 at 10:53 UTC

    Interestingly, bing is the one which fails for me:

    $ cat ssl.t
use strict;
use warnings;

use Test::More;
use LWP::UserAgent;

my @hostnames = (
    'google.com',
    'microsoft.com',
    'bing.com'
);

plan tests => 1 + scalar @hostnames;

cmp_ok ($LWP::UserAgent::VERSION, '>=', 6.04, 'LWP is at least fairly 
+recent');
my $ua = LWP::UserAgent->new;
for my $host (@hostnames) {
    my $res = $ua->head ("https://$host/");
    is ($res->code, '200', "$host returns 200")
}
$ prove -v ssl.t
ssl.t .. 
1..4
ok 1 - LWP is at least fairly recent
ok 2 - google.com returns 200
ok 3 - microsoft.com returns 200
not ok 4 - bing.com returns 200

#   Failed test 'bing.com returns 200'
#   at ssl.t line 19.
#          got: '405'
#     expected: '200'
# Looks like you failed 1 test of 4.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/4 subtests 

Test Summary Report
-------------------
ssl.t (Wstat: 256 Tests: 4 Failed: 1)
  Failed test:  4
  Non-zero exit status: 1
Files=1, Tests=4,  2 wallclock secs ( 0.04 usr  0.02 sys +  0.33 cusr 
+ 0.02 csys =  0.41 CPU)
Result: FAIL

    [download]

    405 is "Method Not Allowed" so bing is just disallowing HEAD requests. I am unable to reproduce your errors and am interested to see your code which works in successfully retrieving a response to a HEAD from bing.com.

    Addendum: here are the relevant version numbers:

    • Perl: 5.16.3
    • LWP: 6.15
    • IO::Socket::SSL: 2.012
    • Mozilla::CA: 20141217
[reply]
[d/l]
      You're right. My code returns 500,500,405 for Bing. But if I change code: 
      use strict;
use warnings;

use LWP::UserAgent;

my @hostnames = (
        'google.com',
        'microsoft.com',
        'bing.com'
);

my $ua = LWP::UserAgent->new;
for my $host (@hostnames) {
#        my $res = $ua->head ("https://$host/");
        my $res = $ua->get ("http://$host/");
        print $res->code . "\n";
}

      [download]
      it returns: 200,500,200
      • CentOS release 6.9 (Final)
      • This is perl, v5.10.1 (*) built for i386-linux-thread-multi
      • 5.833 - LWP
      • none
      • none
[reply]
[d/l]
        500 Can not connect to domen: 443 (connect: Network is unreachable); Client-Warning: Internal response

        Your requests never go outside of your local network if at all. Maybe you need to use a proxy like your browser does, or you have two routes and the name resolution ping-pongs between a working network route and some other network route.

        Maybe the name resolution itself is wonky for some domain names.

[reply]

        When I make the same 2 changes which you make (https to http and head to get) then all my tests pass. If you are unable to make an HTTP connection to http://microsoft.com/ then that has to be a question for your local network administrator.

[reply]

Back to Seekers of Perl Wisdom