Re: LWP::Simple::get($url) does not work for particular urls (SSL certificate verification)

Hi, using getprint() instead shows the error:

500 Can't connect to www.cryptopia.co.nz:443 (certificate verify faile
+d) <URL:https://www.cryptopia.co.nz/api/GetCurrencies>
[download]

( ... because, as the doc for LWP::Simple states:

"The get() function will fetch the document identified by the given URL and return it. It returns undef if it fails."

While for getprint():

"If the request fails, then the status code and message are printed on STDERR. The return value is the HTTP response code."

)

You should use the full LWP::UserAgent so you can disable SSL certificate checking, or use HTTP::Tiny, which has it disabled by default:

use strict; use warnings;
use HTTP::Tiny;
use JSON;
use Data::Dumper;

my $url = "https://www.cryptopia.co.nz/api/GetCurrencies";
my $res = HTTP::Tiny->new->get( $url );
my $decoded_json = decode_json( $res->{'content'} );

print Dumper( $decoded_json );

__END__
[download]

Partial output:

$ perl 1206484.pl | head -20

$VAR1 = {
          'Message' => undef,
          'Error' => undef,
          'Data' => [
                      {
                        'Status' => 'OK',
                        'StatusMessage' => undef,
                        'DepositConfirmations' => 20,
                        'Algorithm' => 'POS',
                        'Name' => '1337',
                        'MaxWithdraw' => '90000000',
                        'MinTip' => '166.66666666',
                        'Symbol' => '1337',
                        'Id' => 331,
                        'ListingStatus' => 'Active',
                        'IsTipEnabled' => bless( do{\(my $o = 0)}, 'JS
+ON::PP::Boolean' ),
                        'WithdrawFee' => '0.01',
                        'MinWithdraw' => '20000',
                        'MinBaseTrade' => '2e-05'
                      },
[download]

Hope this helps!

The way forward always starts with a minimal test.

Comment on Re: LWP::Simple::get($url) does not work for particular urls (SSL certificate verification) Select or Download Code

Replies are listed 'Best First'.
Re^2: LWP::Simple::get($url) does not work for particular urls (SSL certificate verification) by noxxi (Pilgrim) on Dec 31, 2017 at 20:46 UTC
>You should use the full LWP::UserAgent so you can disable SSL certificate checking, or use HTTP::Tiny, which has it disabled by default: Certificate validation is there for a reason and simply recommending to switch it off is essentially suggesting to abandon any security provided by https since man in the middle attacks are easy if proper certificate validation is not done. The real reason that the code fails is a broken setup of the target site. This can be worked around in a secure way by setting SSL_ca_file to the appropriate CA certificates or by using SSL_fingerprint. For details see Stackoverflow where the question was also asked and I've answered it in detail.	[reply]
Re^2: LWP::Simple::get($url) does not work for particular urls (SSL certificate verification) by ssara (Acolyte) on Dec 31, 2017 at 15:48 UTC
Thanks a lot.I have one query regarding valid certificate.If there was issue of SSL certificate then why i am directly able to see the data in web browser when i browse the link https://www.cryptopia.co.nz/api/GetCurrencies in the web browser?	[reply]
Re^3: LWP::Simple::get($url) does not work for particular urls (SSL certificate verification) by 1nickt (Canon) on Dec 31, 2017 at 16:19 UTC
If there was issue of SSL certificate then why i am directly able to see the data in web browser when i browse the link https://www.cryptopia.co.nz/api/GetCurrencies in the web browser? Hi, this is because your desktop browser uses the Certificate Authority root certificates that are distributed with your operating system, whereas LWP::UserAgent does not know about them. You can force LWP to use the same root certificates by setting the value of the environment variable `HTTPS_CA_DIR` to the location on your system. On my system the certificate directory is `/etc/ssl/certs`. Without: `use strict; use warnings; use LWP::UserAgent; use JSON; use Data::Dumper; my $url = "https://www.cryptopia.co.nz/api/GetCurrencies"; my $res = LWP::UserAgent->new->get( $url ); die $res->content if not $res->is_success; print Dumper decode_json $res->decoded_content; __END__` [download] Output: `$ perl 1206484.pl \| head -20 Can't connect to www.cryptopia.co.nz:443 (certificate verify failed) SSL connect attempt failed error:14090086:SSL routines:ssl3_get_server +_certificate:certificate verify failed at /home/nick/perl5/perlbrew/p +erls/perl-5.26.1/lib/site_perl/5.26.1/LWP/Protocol/http.pm line 46.` [download] Now with the UA pointed at the CA cert store: `use strict; use warnings; use LWP::UserAgent; use JSON; use Data::Dumper; BEGIN { $ENV{'HTTPS_CA_DIR'} = '/etc/ssl/certs'; } my $url = "https://www.cryptopia.co.nz/api/GetCurrencies"; my $res = LWP::UserAgent->new->get( $url ); die $res->content if not $res->is_success; print Dumper decode_json $res->decoded_content; __END__` [download] Output: $ perl 1206484.pl \| head -20 $VAR1 = { 'Message' => undef, 'Success' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ), 'Error' => undef, 'Data' => [ { 'MaxWithdraw' => '90000000', 'IsTipEnabled' => bless( do{\(my $o = 0)}, 'JS +ON::PP::Boolean' ), 'DepositConfirmations' => 20, 'Status' => 'OK', 'MinWithdraw' => '20000', 'Name' => '1337', 'Algorithm' => 'POS', 'Symbol' => '1337', 'MinBaseTrade' => '2e-05', 'StatusMessage' => undef, 'MinTip' => '166.66666666', 'ListingStatus' => 'Active', 'Id' => 331, 'WithdrawFee' => '0.01' [download] Hope this helps! The way forward always starts with a minimal test.	[reply] [d/l] [select]
Re^4: LWP::Simple::get($url) does not work for particular urls (SSL certificate verification) by noxxi (Pilgrim) on Dec 31, 2017 at 20:51 UTC
> ... this is because your desktop browser uses the Certificate Authority root certificates that are distributed with your operating system, whereas LWP::UserAgent does not know about them. The problem is not that the root CA is not trusted by LWP but that the intermediate CA is missing (broken server setup) and thus the trust chain cannot be created. Desktop browsers often successfully work around such broken setups by using cached certificates or downloading missing CA from the web but other clients (Python, Perl, Java, mobile apps...) don't do this workarounds and thus fail with broken sites.	[reply]
Re^5: LWP::Simple::get($url) does not work for particular urls (SSL certificate verification) by 1nickt (Canon) on Dec 31, 2017 at 22:16 UTC
Re^4: LWP::Simple::get($url) does not work for particular urls (SSL certificate verification) by ssara (Acolyte) on Dec 31, 2017 at 17:03 UTC
Thanks a lot for the information.	[reply]
Re^3: LWP::Simple::get($url) does not work for particular urls (SSL certificate verification) by choroba (Cardinal) on Dec 31, 2017 at 16:13 UTC
Maybe your browser uses different certificates than your program? ($q=q:Sq=~/;[c](.)(.)/;chr(-\|\|-\|5+lengthSq)`"S\|oS2"`map{chr \|+ord }map{substrSq`S_+\|`\|}3E\|-\|`7**2-3:)=~y+S\|`+$1,++print+eval$q,q,a, [download]	[reply] [d/l]