in reply to Building a Parallel Robot

Hello

You say:

"... that's a bit slow really, I can only visit about a few hundred URLs an hour."

What is your connection speed? How big is the average size of each URL? If you're on a 56kbits/s modem, and the average website is 56kbytes (including headers and other overhead), then you can get a URL/8secs, or about 450 URLs/hour and that should not be slow considered.

Now if you (like me) can get 5MBits/sec, then that would be another story.