WWW::Mechanize doesn't actually do any socket work. It lets LWP do it, so nothing needs to be done. Keep in mind that Coro is cooperative multitasking, so your sockets can't receive anything if your crawler is spending a lot of time not waiting for data.
| [reply] |
Sounds great! So, we just replace the LWP module with your AnyEvent::HTTP / Coro version and things should work for Mechanize out of the box?
Not sure I see what you mean by your point about Coro. If my crawler isn't spending any time waiting for data I will be extremely happy that it is crawling as fast as my network connection allows.
| [reply] |
Not sure I see what you mean by your point about Coro.
Coro isn't threaded! (Despite the blatant lies in the documentation!).
It is cooperative task-switching--like Windows 3.1--which means that if one of your coro instances is busy, none of the others will do anything at all until it either: finishes; goes into a wait for IO; or yields.
It also means that regardless of how many cores you have, it will only ever use one of them.
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply] |