in reply to Crawling all urls on a site

A recent thread about this here: Creating a web crawler (theory).

In Section Seekers of Perl Wisdom