Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi All

I tried to grab a page using this code on my Mint10 PC. For some reason it chokes. I tried the same on another Mint17 PC connected through the same network and it worked. But I don't understand what the error is trying to tell me. Any suggestions? I was able to get python to get the same page on the Mint10 PC too using python's urilib.

Here's the code

    use LWP::UserAgent;
    use HTTP::Request::Common qw(GET);
    use HTTP::Cookies;

    my $ua = LWP::UserAgent->new;

    # Define user agent type
    $ua->agent('Mozilla/8.0');

    # Cookies
    $ua->cookie_jar(
        HTTP::Cookies->new(
            file => 'mycookies.txt',
            autosave => 1
        )
    );

    # Request object
    my $req = GET 'http://www.perlmeme.org';

    # Make the request
    my $res = $ua->request($req) || die "Bad Get";

    # Check the response
    if ($res->is_success) {
        print $res->content;
    } else {
        print ">> ",$res->status_line . "\n";
    }

[download]

here's the response

try_web_get.pl
>> 500 Unexpected keys - LocalHost

[download]

Thanks to anyone who may help

Replies are listed 'Best First'.
Re: LWP::UserAgent error Unexpected keys - LocalHost
by Anonymous Monk on Jul 29, 2014 at 22:50 UTC
    I used your script without any changes and it worked (downloaded perlmeme.org and perlmeme.org/tutorials/lwp.html)

    try this:

    ... # as before...

use Data::Dumper;

my $req = GET 'http://www.perlmeme.org';
print Dumper $req->{_headers};

...

if ($res->is_success) {
    print $res->content;
} else {
    print Dumper $res->{_headers};
}

    [download]
    Also, try Console::Logger. Maybe it will tell something.
[reply]
[d/l]

      Thanks

      I tried the extra lines, this is what I got

      
$VAR1 = bless( {}, 'HTTP::Headers' );
$VAR1 = bless( {
                 'client-warning' => 'Internal response',
                 'client-date' => 'Tue, 29 Jul 2014 23:07:18 GMT',
                 'content-type' => 'text/plain'
               }, 'HTTP::Headers' );
>> 500 Unexpected keys - LocalHost

      [download]

      As I said at the start, I'm not surprised that you were able to make it work, I can too an another PC.

      My problems is that the Mint10 PC that I'm normally using is giving the problem. I'm trying to find out how to fix it. Unfortunately the debugging info is rather hard to get to grips with for a novice.

[reply]
[d/l]
        Hmm... that's not actually LWP's response (despite that 'internal' stuff). Do you use a proxy by any chance? It seems Python's urllib can automatically detect it. Try 
        my $ua = ... # as before
$ua->env_proxy;
...

        [download]
[reply]
[d/l]
Re: LWP::UserAgent error Unexpected keys - LocalHost
by Anonymous Monk on Jul 29, 2014 at 20:34 UTC

    Just say not to typing effort http://www.perlmeme.org/tutorials/lwp.html

    use WWW::Mechanize, its everything you wanted LWP to be

    Same program with less typing 

    #!/usr/bin/perl --
use strict; use warnings;
use WWW::Mechanize 1.73;
my $ua = WWW::Mechanize->new( autocheck => 1 );
$ua->show_progress(1);
$ua->get( 'http://perl-tutorial.org/' );
print $ua->content;
__END__
** GET http://perl-tutorial.org/ ==> 200 OK (1s)
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
...

    [download]

    Also, upgrade LWP, like with cpanm LWP

[reply]
[d/l]
      Hi,

      sorry to say, but Mechanize uses LWP::UserAgent under the bonnet as far as I can gather. So the net result remains the same.

      use WWW::Mechanize;
my $mech = WWW::Mechanize->new( autocheck => 1 );

$mech->get($url);

if ( $mech->success() ) {
    print $mech->content,"\n";
}
else {
    print "status is: " . $mech->status,"\n";
}

      [download]

      Returns

      try_web_again.pl
Error GETing http://perlmeme.org/tutorials/lwp.html: Unexpected keys -
+ LocalHost at /home/bill/bin/try_web_again.pl line 15

      [download]

      Also, according to cpan...

      Database was generated on Tue, 29 Jul 2014 06:06:23 GMT All modules are up to date for LWP

      So it seems thast I have the latest LWP

      regards.

[reply]
[d/l]
[select]

        sorry to say, but Mechanize uses LWP::UserAgent under the bonnet as far as I can gather.

        Yes, thats the point, its less typing. Autocheck dies on errors, like you witnessed. No setting up a cookiejar, no saving responses... less typing is less typing :)

        So it seems thast I have the latest LWP ..

        That isn't exactly convincing :) LWP has many dependencies, upgrade all of them :)

        You could tty  perl -MCarp::Always yourprog.pl for a stack trace to narrow down which IO::Socket... module is complaining, which module needs specific updating

        Probably IO::Socket::IP and IO::Socket::SSL

[reply]
[d/l]
Re: LWP::UserAgent error Unexpected keys - LocalHost
by ikegami (Patriarch) on Jul 30, 2014 at 16:32 UTC
    Sounds like one of the modules used by LWP was updated without having its dependencies updated to the new minimum required version. Could you please provide the output of the following command: 
    perl -MLWP::UserAgent -e'
   END {
      for (keys(%INC)) {
         s{\.pm\z}{};
         s{/}{::}g;
         printf("%s %s\n", $_, $_->VERSION);
      }
   }

   LWP::UserAgent->new->get("http://www.perlmeme.org")
' | sort

    [download]
[reply]
[d/l]
      Sorry, forgot to reply here.

      FYI, here's the output to the above command.

      
Can't call method "VERSION" without a package or object reference at -
+e line 6.
END failed--call queue aborted.
Fcntl 1.06
HTTP::Request::Common 6.00
HTTP::Status 6.00
IO::Handle 1.28
IO::Socket 1.31
LWP::UserAgent 6.06
Net::HTTP 6.07
POSIX 1.17
SelectSaver 1.02
Symbol 1.07
Tie::Hash 1.03
Time::Local 1.2000
URI::_generic 
warnings 1.06

      [download]
[reply]
[d/l]

        Oh, and here's the same from the new ActiveState install

        
perl -MLWP::UserAgent -e'
   END {
      for (keys(%INC)) {
         s{\.pm\z}{};
         s{/}{::}g;
         printf("%s %s\n", $_, $_->VERSION);
      }
   }

   LWP::UserAgent->new->get("http://www.perlmeme.org")
' | sort
bytes 1.04
Carp 1.32
Compress::Raw::Zlib 2.063
Config 
constant 1.27
Errno 1.15
Exporter 5.68
Exporter::Heavy 5.68
Fcntl 1.11
feature 1.27
File::Glob 1.17
File::GlobMapper 1.000
HTML::Entities 3.69
HTML::HeadParser 3.71
HTML::Parser 3.71
HTTP::Config 6.00
HTTP::Date 6.02
HTTP::Headers 6.05
HTTP::Message 6.06
HTTP::Request 6.00
HTTP::Request::Common 6.04
HTTP::Response 6.04
HTTP::Status 6.03
integer 1.00
IO 1.25_06
IO::Compress::Base::Common 2.063
IO::Compress::Gzip::Constants 2.063
IO::Compress::Zlib::Extra 2.063
IO::File 1.16
IO::Handle 1.33
IO::Seekable 1.1
IO::Select 1.21
IO::Socket 1.34
IO::Socket::INET 1.33
IO::Socket::UNIX 1.24
IO::Uncompress::Adapter::Inflate 2.063
IO::Uncompress::Base 2.063
IO::Uncompress::Gunzip 2.063
IO::Uncompress::RawInflate 2.063
List::Util 1.35
LWP 6.05
LWP::MemberMixin 
LWP::Protocol 6.00
LWP::Protocol::http 
LWP::UserAgent 6.05
Net::HTTP 6.06
Net::HTTP::Methods 6.06
overload 1.18
overloading 0.02
Scalar::Util 1.35
SelectSaver 1.02
Socket 2.013
Storable 2.45
strict 1.07
Symbol 1.07
Time::Local 1.2300
URI 1.60
URI::Escape 3.31
URI::_generic 
URI::http 
URI::_idna 
URI::_punycode 0.04
URI::_query 
URI::_server 
utf8 1.09
vars 1.02
warnings 1.13
warnings::register 1.02
XSLoader 0.16

        [download]
[reply]
[d/l]
Re: LWP::UserAgent error Unexpected keys - LocalHost
by Khen1950fx (Canon) on Jul 30, 2014 at 01:52 UTC
    Here's a slightly modified script that worked on my versions of perl from 5.8.8 thru 5.20.0. Note: Always use strictures. 
     
#!/usr/bin/perl -l    

use strict;
use warnings;
require LWP::UserAgent;

my $ua = LWP::UserAgent->new;
$ua->agent("Mozilla/8.0");
$ua->cookie_jar(
    {   file => 'mycookies.txt',
        autosave => 1
    }
);

my $url = 'http://www.perlmeme.org/';
my $req = HTTP::Request->new( 'GET', $url );
my $res = $ua->request( $req, undef, undef ) || die( 'Get'->Bad );

if ( $res->is_success ) {
    print $res->decoded_content;
}
else {
    print ">>  ", $res->status_line;
}

    [download]
[reply]
[d/l]

      Here's a slightly modified script that you can ignore

      That won't make any difference

      The OPs IO::Socket::SSL/IO::Socket::IP... and other supporting LWP modules are out of sync (one is new other is old)

      OP needs the latest ones and to learn to ignore irrelevant advice

[reply]

        Thanks for the suggestions. I still cannot figure what tweak would fix it for me.

        So I've taken the pragmatic approach and installed a parallel 'ActiveState' perl.

        It has all the right modules pre-installed. And LO, all the tests that failed previously now work with the Active Perl.

        It's a good enough compromise until I get my upgrade PC up and running.

        Regards to all.
[reply]

Back to Seekers of Perl Wisdom