in reply to Crawling all urls on a site

Try searching for a web spider. That should get you going. However, you do need to pay attention to the fact that not all pages want to be crawled. A good spider will honor those requests.

Useless trivia: In the 2004 Las Vegas phone book there are approximately 28 pages of ads for massage, but almost 200 for lawyers.