in reply to Need spidering solution

Merlyn has several columns on that. Here's one of them.