Along these lines, you might find that using WWW::Robot decreases your runtime, by returning only those pages that the site wishes to have spidered.