in reply to Re: Re: suck on www::mechanize question
in thread can't get www::mechanize to work on a web site

OK, further investigation suggests that it's not one bad link, but that the server just sometimes returns a 500 error. Other times it succeeds. If I run my script several times, it will sometimes succeed and sometimes fail -- and I get the same results if I try to follow the link several times in my browser.

I was able to get around it like this: 

my $maxtries = 10;
my $i = 1;
while ($i <= $maxtries) {
    $mech->follow_link(text => "Hey Jude (lennon/mccartney)", n => "1"
+);
    last if $mech->success;
    $mech->back();
    $i++;
}

$mech->success() or die "Can't find song page\n";

[download]

Of course, you can set $maxtries to whatever you think is prudent, and you can put in a sleep() in the loop if you think that might help.

Linda

Replies are listed 'Best First'.
STUCK on www::mechanize ->get()
by smackdab (Pilgrim) on Mar 01, 2004 at 22:13 UTC
    Thanks again for the help

    I tried your string 200 times w/o success and it works every time in the browser...Don't know what I could be doing wrong...

    I don't think it is the lyrics web site as the first ->get() works, it is just the 2nd one that fails...

    The link that failes in www:mechanize is:

    http://display.lyrics.astraweb.com:2000/display.cgi?beatles..beatles_1..hey_jude

    This URL works in my browser always...

    Here is my complete test program incase someone else wants to give it a try ;-)

    use WWW::Mechanize;
use URI::URL;
use strict;  use warnings;

my $artist = 'The Beatles';
my $title  = 'Hey Jude';

my $mech   = WWW::Mechanize->new(autocheck=>1);
#$mech->agent_alias('Windows IE 6');

my $search = join("+", split(/ /, $artist)) . "+" . join("+", split(/ 
+/, $title));
print "search=" .  "http://search.lyrics.astraweb.com/?word=$search" .
+ "\n";
$mech->get("http://search.lyrics.astraweb.com/?word=$search");
$mech->success() or die "Can't get the search page\n";
$mech->follow_link(text=>"Hey Jude");
#$mech->follow_link(n=>6);
$mech->success() or die "Can't find lyrics page\n";

    [download]
[reply]
[d/l]
      Your script works for me as often as following the like works for me in a browser.

      I'd suggest that you try to get some more information about what's going on. Instead of just die-ing after a failure, you might try printing out $mech->status and/or $mech->content to see if that sheds any light on what's going on.

      Sorry I can't be of more help.

      Linda

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom