merrymonk has asked for the wisdom of the Perl Monks concerning the following question:

On Jan 28, 2018 at 21:25 UTC ( #1208014=perlquestion: print w/replies, xml ) I asked a question about reading (UK) BBC 4 Radio schedules.

Monk tangent very helpfully replied giving Perl code which solved the problem. The Perl used LWP::UserAgent and is as follows
use LWP::UserAgent;
my $url = 'http://www.bbc.co.uk/radio4/programmes/schedules/fm/2015/10
+/13';
my $ua = LWP::UserAgent->new(ssl_opts => {verify_hostname => 0});
my $res = $ua->get( $url );
my $html = $res->content;
print $html;

[download]

I am using an implementation of Perl which is supplied by a commercial organization so that I can run their application based on Perl.
Therefore I have no detailed knowledge about how to add facilities such as LWP::UserAgent.
However, I did try but found I after I had added a Perl module yet another Perl module was required. For example, when using the Perl supplied by tangent I now get the following message:
“LWP will support https URLs if the LWP::Protocol::https module is installed.”

I did not get to a successful conclusion.

I have a new set of Perl installed so that all that I added (very likely incorrectly) is no longer there. Therefore I want to try again.

Can a monk give me the steps so that I can do whatever is necessary to get LWP::UserAgent (taking into account the message I got when trying the Perl sent by tangent) and any required Perl modules added to my version of Perl?

Replies are listed 'Best First'.
Re: BBC4 Radio Schedules and LWP:UserAgent problem
by hippo (Archbishop) on Apr 07, 2018 at 13:51 UTC

    You do not need to specify ssl_opts if your URL is not https. That way your code will run without many of the SSL-specific dependencies.

    For how to install modules, start with A Guide to Installing Modules. If you want to see the full tree of dependencies, take a look at http://deps.cpantesters.org/

[reply]
[d/l]

      The http://www.bbc... URL redirects (301 code - permanently) to an https://www.bbc... URL, so he likely does need SSL.

      Ron
[reply]
      Thank your for the links about installing modules and getting dependencies. From your comments, I tried modifying the line that 'called' UserAgent to
      
my $ua = LWP::UserAgent->new();

      [download]
      However, I again got the same error message. I suspect I did not do all that is required and I regret to say I do not understand the implication of your comment "That way your code will run without many of the SSL-specific dependencies."
      I would appreciate some more clues about what I should do.
      The first lines of the 'page-source' of the URL in the Perl test code is next - it seems to be HTML
      <!DOCTYPE html>
      <html class="b-header--black--white b-footer--black--white " lang="en-GB">
      <head>
      I am more than happy to try and load the required Perl modules but I am struggling to know where to start.
[reply]
[d/l]
        I do not understand the implication of your comment "That way your code will run without many of the SSL-specific dependencies."

        So, LWP::UserAgent is there to give you a client for accessing web resources which may be reached either via HTTP or HTTPS. The latter requires SSL (or TLS) and those require lots of extra code in the form of crypto libraries and so forth. That's what your error message is talking about. If the only web resources you are trying to access are over HTTP then you don't need those extra libraries, modules and so on. Note that I'm grossly simplifying here to keep it understandable.

        However, while the sample code you provided lists only an HTTP URL, mr_ron has pointed out that this merely redirects to an HTTPS URL and therefore you do in fact need all the extra code in order to get to the end resource which requires HTTPS. Still with us?

        Now, here's some sample code using the real, end-point URL explicitly:

        #!/usr/bin/env perl 
use strict;
use warnings;
use LWP::UserAgent;

my $url = 'https://www.bbc.co.uk/radio4/programmes/schedules/fm/2015/1
+0/13';
my $ua = LWP::UserAgent->new ();
my $res = $ua->get( $url );
my $html = $res->content;
print substr ($html, 0, 256) . "...\n";

        [download]

        which produces this output:

        $ perl getr4.pl 
<!DOCTYPE html>
<html class="b-header--black--white b-footer--black--white " lang="en-
+GB">
    <head>
        <meta charset="UTF-8">
        <title>BBC Radio 4 FM - Schedules, Tuesday 13 October 2015</ti
+tle>
        <link rel="icon" href="https://www.bbc.c...
$

        [download]

        This is using perl 5.20.3 and LWP::UserAgent 6.15, Mozilla::CA 20141217, LWP::Protocol::https 6.06, There are alternatives, but you can start with these. Try installing suitably recent versions of these modules using the documentation you have already read. You may need to install other dependencies too. Good luck.

[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom