One thing to consider with scraping webpages, whether you use mechanize or not. Is that many servers just "barf" on some requests. That's what is it - I try to design my servers so that they don't do that - but not every body does.

When you are the human, you click again and this one out of 2,000 requests just doesn't even register in your brain. If I have have to run 8,000 requests, then it matters...

Here is some code that you can adapt:

You should do a "retry" before deciding that this is a "dead link". I show one way below. This server barfs with error 500 or whatever about 1/2000 requests.

The RETRY skips the (while) statement and continues on with a new GET. I don't bother to "skip around" the "clean-up" code before the GET because it runs really fast and again only happens 1/2000 times.

Hope this idea helps you. This is real world stuff that does happen. I sleep a little bit to "be nice". This code works with a "paid subscription" and I am not as "nice" as I would be if this was a free interface. but even so I am a little nice when the server "barfs".

The main point here is the use of RETRY: (which is my label) and redo which is the Perl keyword. 

RETRY: while (my $n_attempt=0, my $callsign=<>)
{
    $callsign =  uc($callsign); # uppercase
    $callsign =~ s/^\s*//;      # no leading spaces
    $callsign =~ s/\s*$//;      # no trailing spaces does chomp() also
+..
    next if $callsign eq "";    # skip NULL (blank lines)!
   
    my $callsign = (split(/,/,$callsign))[0]; #allow histogram format
                                              #w6oat,234 or just w6oat
                                             
    next if ($callsign =~ /^[a-zA-Z]\d{1}[a-zA-Z]$/); # like N7A 
    # NO PROCESSING OF 1X1 US CALLSIGNS!!!
   
    print STDERR "working on $callsign\n" if DEBUG;
    
    my $req = GET "http://www.qrz.com/xml?s=$key;callsign=$callsign";
    my $res = $ua->request($req);
   
    unless ($res->is_success)
    {
        $n_attempt++;
        print STDERR "$callsign ERROR: Try# $n_attempt of ".MAX_RETRY.
                     " err:". $res->status_line ."\n";
        sleep(1);
        redo RETRY if $n_attempt <= MAX_RETRY;
      
        print STDERR "$callsign ERROR: Try# $n_attempt of ".
                      MAX_RETRY." FAILED: ".
                      $res->status_line . "\n";  
                    
        next;   # skip this callsign and go to the next one.
                # This ain't gonna happen unless the QRZ server is
                # down. "if ($res->is_success)" means we got some kind
                # of response from the server. The QRZ server will
                # barf on about 1/2000 requests, hence the retries.   
+             
    }

[download]

In reply to Re: conditional testing for error 500 webpages before following them? by Marshall
in thread conditional testing for error 500 webpages before following them? by fraizerangus

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.