in reply to Building a Parallel Robot
You say:
"... that's a bit slow really, I can only visit about a few hundred URLs an hour."
What is your connection speed? How big is the average size of each URL? If you're on a 56kbits/s modem, and the average website is 56kbytes (including headers and other overhead), then you can get a URL/8secs, or about 450 URLs/hour and that should not be slow considered.
Now if you (like me) can get 5MBits/sec, then that would be another story.
|
|---|