I have a program design question for the monks today.

I'm working on a web-spidering application using Win32::IE::Mechanize to do the spidering and HTML::Treebuilder to extract information. I'm running into difficulty dealing with intermittent network issues -- sometimes for a few seconds I can't reach the remote site, or the site is unresponsive, or something else impedes the flow of the program.

This is, as we all know, a fact of life on the internet. But at the minute, if my program doesn't find the piece of information it's looking for on the page it thinks it loaded, it dies.

Now part of the problem is that I'm not checking for errors robustly enough. I'm about to put just such error checking in.

But another part of the problem is that I'd like to be able to catch such a failure and retry a couple of times before finally giving up. I'm having trouble coming up with an elegant architecture for doing this, and I would love some advice from some the rest of you wise monks on how to do this nicely. The ideal solution would also be abstract enough to use easily with multiple sequences of actions -- I have a couple of different programs that all face the same issue.

Thanks very much for your meditation on my difficulty.

In reply to catching failures and retrying by mickey

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.