Thanks ikegami for your suggestions and code. Quite a few pointers for improvements there. I was aware of the need to respect robots.txt so you have helped me there too.

Thanks again.