in reply to Crawling all urls on a site

Should you change your mind about the use of other modules, take a look at WWW::Mechanize also...