in reply to Re: Infinite loop prevention for spider
in thread Infinite loop prevention for spider

The solution, then, is to start spidering with a large inventory of items (ie, a shovel, perhaps some miscellaneous treasure). As you spider each page, drop one of your inventory items into that page. Then when you visit a page again, you can tell which one it is by which inventory item is there.

Oh, and make sure your spider has a lantern, or else it is likely to be eaten by a grue...

  • Comment on Re: Re: Infinite loop prevention for spider

Replies are listed 'Best First'.
A reply falls below the community's threshold of quality. You may see it by logging in.