nachtmsk has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I need to get JSON data from a URL. This is the URL, minus the actual Auth code at the end.

https://api.bridgedataoutput.com/api/v2/OData/test/TestData?access_token=xxxxxxxxxxxxxxxxxxxxxx

If I type this URL into my browser it brings back the data I need. However, if I use it in the code below I get the 'die' message I have set below. My main task right now is to just get the data from the URL into a perl variable. Later on I can worry about parsing the JSON. I have used LWP before to get data this way, usually XML. I can't figure out why this isn't working.

Any help would be appreciated.

--------------------------- 
use strict;
use warnings 'all';

use LWP::Simple 'get';
use JSON::Parse 'parse_json';

use constant URL => 'https://api.bridgedataoutput.com/api/v2/OData/tes
+t/TestData?access_token=xxxxxxxxxxxxxxxxxxxxxx';

my $json = get URL or die "Unable to get JSON data";
my $data = parse_json($json);

print $data;

[download]
Thanks! Mike

Replies are listed 'Best First'.
Re: Getting JSON data from a URL using LWP
by choroba (Cardinal) on Jan 23, 2020 at 21:08 UTC
    Have you installed LWP::Protocol::https? Without it, LWP::Simple can only handle http:// URLs.

    map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]
[reply]
[d/l]
      I don't have LWP::Protocol::https installed. Didn't even notice it was an https url. Trying to install it now. Giving me an error with LWP-Protocol-https-6.07.tar.gz. Working on it... Thanks! Mike
[reply]

      Do I need to buy an SSL cert and install on my machine for the LWP::Protocol::https mod to install? I didn't think so, it looks like the installer is creating it's own cert. I'm having some issues getting it installed. The error messages aren't telling me enough about what is wrong. Thoughts? 

      cpan[5]> install LWP::Protocol::https
Running install for module 'LWP::Protocol::https'
Running make for O/OA/OALDERS/LWP-Protocol-https-6.07.tar.gz
Checksum for /var/root/.cpan/sources/authors/id/O/OA/OALDERS/LWP-Proto
+col-https-6.07.tar.gz ok

  CPAN.pm: Building O/OA/OALDERS/LWP-Protocol-https-6.07.tar.gz

Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for LWP::Protocol::https
Writing MYMETA.yml and MYMETA.json
cp lib/LWP/Protocol/https.pm blib/lib/LWP/Protocol/https.pm
Manifying 1 pod document
  OALDERS/LWP-Protocol-https-6.07.tar.gz
  /usr/bin/make -- OK
Running make test
PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::H
+arness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/l
+ib', 'blib/arch')" t/*.t
t/apache.t ....... Argument "" isn't numeric in bitwise or (|) at /Lib
+rary/Perl/5.18/IO/Socket/SSL.pm line 2254.
t/apache.t ....... 1/5 
#   Failed test at t/apache.t line 15.

#   Failed test at t/apache.t line 18.
#                   'Can't connect to www.apache.org:443 (SSL connect 
+attempt failed error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tls
+v1 alert protocol version)
# 
# SSL connect attempt failed error:1407742E:SSL routines:SSL23_GET_SER
+VER_HELLO:tlsv1 alert protocol version at /Library/Perl/5.18/LWP/Prot
+ocol/http.pm line 50.
# '
#     doesn't match '(?^:Apache Software Foundation)'

#   Failed test at t/apache.t line 24.
# Looks like you failed 3 tests of 5.
t/apache.t ....... Dubious, test returned 3 (wstat 768, 0x300)
Failed 3/5 subtests 
t/https_proxy.t .. 1/56 # creating cert for direct.ssl.access
# creating cert for direct.ssl.access
unexpected response: 500 SSL upgrade failed: Client side SNI not suppo
+rted for this openssl
Content-Type: text/plain
Client-Date: Thu, 23 Jan 2020 21:47:54 GMT
Client-Warning: Internal response

SSL upgrade failed: Client side SNI not supported for this openssl at 
+/Library/Perl/5.18/LWP/Protocol/http.pm line 208.
# creating cert for foo
# Looks like your test exited with 25 just after 36.
t/https_proxy.t .. Dubious, test returned 25 (wstat 6400, 0x1900)
Failed 20/56 subtests 

Test Summary Report
-------------------
t/apache.t     (Wstat: 768 Tests: 5 Failed: 3)
  Failed tests:  1, 3-4
  Non-zero exit status: 3
t/https_proxy.t (Wstat: 6400 Tests: 36 Failed: 0)
  Non-zero exit status: 25
  Parse errors: Bad plan.  You planned 56 tests but ran 36.
Files=2, Tests=41,  0 wallclock secs ( 0.03 usr  0.01 sys +  0.47 cusr
+  0.05 csys =  0.56 CPU)
Result: FAIL
Failed 2/2 test programs. 3/41 subtests failed.
make: *** [test_dynamic] Error 25
  OALDERS/LWP-Protocol-https-6.07.tar.gz
  /usr/bin/make test -- NOT OK
//hint// to see the cpan-testers results for installing this module, t
+ry:
  reports OALDERS/LWP-Protocol-https-6.07.tar.gz
Running make install
  make test had returned bad status, won't install without force
Failed during this command:
 OALDERS/LWP-Protocol-https-6.07.tar.gz       : make_test NO

cpan[6]> reports OALDERS/LWP-Protocol-https-6.07.tar.gz
Distribution: O/OA/OALDERS/LWP-Protocol-https-6.07.tar.gz
CPAN::DistnameInfo not installed; cannot continue

      [download]
[reply]
[d/l]
        #   Failed test at t/apache.t line 18.
#                   'Can't connect to www.apache.org:443 (SSL connect 
+attempt failed error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tls
+v1 alert protocol version)
# 
# SSL connect attempt failed error:1407742E:SSL routines:SSL23_GET_SER
+VER_HELLO:tlsv1 alert protocol version at /Library/Perl/5.18/LWP/Prot
+ocol/http.pm line 50.

        [download]

        This suggests that you might be running against a very old version of the openssl libs which don't support TLSv1.2 (which is the oldest protocol that particular server supports). Try running openssl version and see which version you have. Maybe it's time to upgrade?

[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom