jonnyfolk has asked for the wisdom of the Perl Monks concerning the following question:

I am using LWP::Simple to grab content from a page, and parse the data for exchange rates.

The url of the page is: https://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml?025a000c2e476c719095c900b9c8b5e4

If I go to the page in my browser I can see the (xml) content. Using the script it is not getting the content. Why might this be? (It used to work!). 

#!/usr/bin/perl -w
use strict;
use CGI::Carp qw(fatalsToBrowser warningsToBrowser);
use CGI ':standard';
use LWP::Simple;

my $address = 'https://www.ecb.europa.eu/stats/eurofxref/eurofxref-dai
+ly.xml?025a000c2e476c719095c900b9c8b5e4';

my $content = get("$address");
die "Couldn't get it!" unless defined $content;

print "Content-type:text/html\n\n"; 
print $content;

[download]

Replies are listed 'Best First'.
Re: LWP::Simple not getting content
by hippo (Archbishop) on Jul 26, 2018 at 12:40 UTC
    Why might this be?

    Could be almost anything, but my guess is on the certificate issuer not being in your CA list. The current cert was only issued on the 7th of June so that's a recent change. What version of Mozilla::CA are you using? It works fine with mine (20180117).

[reply]

      Thanks for your reply. I switched over to LWP::UserAgent and successfully used the following code

      my $url = 'https://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.x
+ml?025a000c2e476c719095c900b9c8b5e4';

my $ua = LWP::UserAgent->new(
  ssl_opts => { SSL_verify_mode => 'SSL_VERIFY_NONE'},
);

   $ua->agent('Mozilla/4.0 (compatible; MSIE 5.12; Mac_PowerPC)' );

my $response = $ua->get($url);
 
if ($response->is_success) {
    print $response->decoded_content;  # or whatever
}
else {
    die $response->status_line;
}

      [download]
[reply]
[d/l]
Re: LWP::Simple not getting content
by Corion (Patriarch) on Jul 26, 2018 at 12:38 UTC

    Maybe the remote server does not like the LWP::Simple useragent header?

    My suggestion is to use something like LWP::ConsoleLogger and to compare and adapt what your script sends with what your browser sends until the two match, or you get a satisfactory result.

[reply]

Back to Seekers of Perl Wisdom