Thanks for the mention of LWP.

I'm currently using the Lynx browser to grab the page before analysing it. Would I gain any advantage from LWP?

I appreciate the need to be aware of robots.txt but the crawler only goes over one site and they have OK'ed the crawling of it. I wasn't aware of a specific module for this though so I'll look into it if I expand this project or begin a new one.

Thanks again everyone, this really is a great place.

Chris.