in reply to Parallel WEB spider, DB support

Threaded solutions are possible, easy and very scalable.

Niceties like not hitting the same server to frequently; only downloading a given url once; not retrying failing servers many times over; are all made easier by shared state.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

In Section Seekers of Perl Wisdom