Most spiders are based around an array used as a queue. You push the first (set of) urls onto the queue and then start the spider, it pulls a url off the queue, fetches it, extracts any links and pushes them onto the queue and loops.

To support retries, instead of pushing just the url, push a url/count pair. Either as an anon array

push @urls, [ $tries, $url ];
[download]

Or you could concatenate them into a string if lots of 2-element arrays proves to be a memory problem.

Preset $tries to 3 or 5 or whatever, and each time you fail, decrement the count and push it back on the queue if it hasn't reached 0 yet. When it reaches 0 give up and report the failure.

while(1) { 
    eval { 
        do_something_that_may_die();
    };
    last unless $@;
    print "Something blew up: $@\nRetrying...";
}
[download]

Thanks very much. Error looks useful for error handling... I think that's a secondary concern for me, though.

The big issue, I think (I'm still working on understanding what the crux of the problem is myself), is how to go back to a previous step.

For instance, if my program proceeds like this:

## Step 1
go_to_web_page_A();

## Step 2
look_for_stuff_on_web_page_A();

## Step 3
submit_form_on_A_to_go_to_B();
[download]

and it dies on step 2, that's almost always because step 1 failed. That is, it can't find the stuff it's looking for because the web page failed to load correctly.

So what I want to do is catch the failure of look_for_stuff_on_web_page_A() and, in case of a failure, retry go_to_web_page_A(); and then look_for_stuff_on_web_page_A(); a couple times before finally giving up.

Basically what it's doing is "Do A; Do B; If B fails, go back and do A again and then retry B;", and the part I'm having trouble with is the "If B fails, go back to A" part.

Any thoughts?

Error provides you a generic and extensible framework to handle errors, but it won't do exactly what you're looking for.

You basically need a method of retrying, which can be handled as BrowserUk states, or, based upon your code - can be something like:

  # CODE NOT TESTED 
  # for display purposes only 
  # void in Utah

  my $rc = undef;
  my $i = 0;
  do {
    $rc = go_to_web_page();
  } 
  while ( !$rc && $i++ < 3 );
[download]

or, perhaps, using Error

  # CODE NOT TESTED 
  # for display purposes only 
  # void in Utah

  my $i = 0;

  GET_PAGE:
  while ( $i < 3 ) {
    try {
      go_to_web_page();
      # the above has to throw some kind of error 
      # for this to work
      last GET_PAGE;
    } 
    catch Error with {
      $i++;
    };
  }
[download]

I'm sure there's a cleaner way to do that, but it's just an example off the top of my head (proof of concept).
--------------
It's sad that a family can be torn apart by such a such a simple thing as a pack of wild dogs

Create a wrapper function (we'll call it wrapper) that returns true on success and false on failure. It might look like this, then:

sub wrapper {
   eval {
      ## do some stuff that might die
   };
   if ($@) {
      ## handle some error things, die if unrecoverable
      return 0;  # returns false if recoverable
   }
   return 1;     #returns true if we should retry
}
[download]

To retry a few times before giving up, then, you may write:

use constant RETRIES => 3;
for (1..RETRIES) { last if wrapper() }
[download]

In this way, you will try up to three times before the loop will exit. If you need more complexity, you might expand to something like:

use constant RETRIES => 3;
our $give_up = 0;
for (1..RETRIES) { 
   $give_up = 0;
   last if wrapper();
   $give_up = 1;
}
die ('Tried '.RETRIES.' times without success. Giving up.')
   if $give_up;
[download]

This will set $give_up on each try in such a way that a sucessful run will result in $give_up == 0, while a failed run will result in $give_up == 1. We can then check that value to see if we gave up or no.

Trouble is, I'm not sure if the Win32 version sets status the same way the base Mech does. If not, you may need to do some more work to evaluate the results of a 'get'.