I would use a hash to store each visited link and use the exists function before visiting a link. The main issues would be:

• You probably want to ensure that you have absolute links (not relative ones), in the hash.
• Ensure that any links that point to scripts have the query string stripped. Rough guess as to the best way to do that:

  $link =~ s/\?.*$//;
  $visited{ $link } = '';
  [download]

Without knowing how you're getting into the infinite loop, I can only offer one other suggestion: for the %visited hash, you may have a problem if a redirection occurs as you'll resolve to the wrong URL. If that's likely to be an issue, you'll have to use LWP::UserAgent and examine the response code.

Cheers,
Ovid

Join the Perlmonks Setiathome Group or just click on the the link and check out our stats.

For what its worth, linkcheck is available on CPAN. I love CPAN. CPAN makes me happy.

A general question gets a general answer. . .
Are you going to be checking static or dynamic pages? This "infinite loop" thing it probably more of an issue for dynamic pages, frequently there are multiple ways of looking at a page (normal, printable, lo-bandwidth, whatever) so you'd need to take this into account.

If you're spidering static pages, what about dumping the address of the page into a DB, and then checking a new page to verify against the DB. If the page exists, then skip it.

If they are dynamic pages, and each document has a unique id of some sort, you could parse out the ID and store that in the DB. Then the multiple views thing would be moot, because the ID is what you're matching on.

update:A hash would probably be better speed wise. With a DB you could have the invalid links stored there and report on them later easily.

$string = 'http://www.whatever.html/directory/';

$string .= 'index.html' if(substr($string, -1) eq '/');
[download]

bbfu
Seasons don't fear The Reaper.
Nor do the wind, the sun, and the rain.
We can be like they are.