kepler has asked for the wisdom of the Perl Monks concerning the following question:

Hi

I'm having some troubles regarding a simple get for an url, https://www.earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_week.atom

Even using this script, I get a null result (in several hosts except one, bluehost): 
#!/usr/bin/perl

use LWP::Simple;
use warnings;

print "Content-type: text/html\n\n";

my $url = "https://www.earthquake.usgs.gov/earthquakes/feed/v1.0/summa
+ry/2.5_week.atom";

my $data = get $url;

print "$url\n$data";

[download]

Can someone help me out?

Thanks, Kepler

Replies are listed 'Best First'.
Re: Get url
by huck (Prior) on Apr 29, 2017 at 21:45 UTC

    And farther on 

    #!/usr/bin/perl
use strict; use warnings; 


  use LWP;
  my $url = "https://www.earthquake.usgs.gov/earthquakes/feed/v1.0/sum
+mary/2.5_week.atom";

  my $ua; 
  $ua = new LWP::UserAgent(keep_alive=>1
                          ,agent =>"Mozilla/5.0 (Windows NT 5.1; rv:51
+.0) Gecko/20100101 Firefox/51.0"
                          ,ssl_opts => { verify_hostname => 0
                                       , SSL_verify_mode => 'SSL_VERIF
+Y_NONE' 
                                       }
#                          ,timeout=>$timeout
                          );
  my $req = new HTTP::Request (GET => $url); 
  my $response = $ua->request ($req);

  unless ($response->is_success) {
     die $response->status_line; 
     }

    [download]
    result 
    404 Not Found at 1189178a.pl line 20.

    [download]

[reply]
[d/l]
[select]

      with my browser (on my phone right now, so no easy perl), I verified that while https gives 404, http gives atom file. kepler should try your error-checking with correct protocol.

[reply]
Re: Get url
by Discipulus (Canon) on Apr 29, 2017 at 21:33 UTC
    hello kepler

    at first glance and using LWP::UserAgent 

    perl -MLWP::UserAgent -e "print LWP::UserAgent->new->get($ARGV[0])->co
+ntent" https://www.earthquake.usgs.gov/earthquakes/feed/v1.0/summary/
+2.5_week.atom

Can't connect to www.earthquake.usgs.gov:443 (certificate verify faile
+d)

LWP::Protocol::https::Socket: SSL connect attempt failed with unknown 
+errorerror:14090086:SSL routines:SSL3_GET_SERVER_C
ERTIFICATE:certificate verify failed at C:/ulisse/strawberry/perl/vend
+or/lib/LWP/Protocol/http.pm line 51.

#another tool same response:

Response code:            500
Response message:         Can't connect to www.earthquake.usgs.gov:443
+ (certificate verify failed)

    [download]

    But telling LWP::UserAgent to not check ssl LWP::UserAgent->new(ssl_opts => { verify_hostname => 0 }) all is fine:

    perl -MLWP::UserAgent -e "print LWP::UserAgent->new(ssl_opts => { veri
+fy_hostname => 0 )->get($ARGV[0])->content" 
https://www.earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_week
+.atom

<!DOCTYPE HTML>
<html>
<head>
  <meta charset="utf-8">
  <title>USGS URL Resolution Error Page</title>
  [...]

    [download]

    L*

    There are no rules, there are no thumbs..
    Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.
[reply]
[d/l]
[select]
Re: Get url
by shmem (Chancellor) on Apr 29, 2017 at 21:39 UTC

    Retrieving this url with firefox gives me:

    Your connection is not secure

    The owner of www.earthquake.usgs.gov has configured their website improperly. To protect your information from being stolen, Firefox has not connected to this website.

    Learn more…

    So it looks like LWP::Simple does https requests with certificate validation turned on by default.

    update: So use something like this:

    perl -MLWP::UserAgent -e 'print LWP::UserAgent->new(ssl_opts => {verif
+y_hostname => 0})->get($ARGV[0])->content' https://www.earthquake.usg
+s.gov/earthquakes/feed/v1.0/summary/2.5_week.atom

    [download]
    perl -le'print map{pack c,($-++?1:13)+ord}split//,ESEL'
[reply]
[d/l]
Re: Get url
by NetWallah (Canon) on Apr 30, 2017 at 01:01 UTC
[reply]

Back to Seekers of Perl Wisdom