Then what technology would you suggest for project like this? Is there any interpreter that has stable production ready support for multithreading? I had a look at ruby and python. Ruby seems like a good compromise between Perl with its extremely rich syntax and Python with its oversimplified syntax. However ruby's port of WWW::Mechanize is in alpha release as well as curb (curl bindings). Also it is new and has very poor documentation in contrast with Perl. However, ruby's syntax is much more clear in contrast with Perl's, Perl has really unsafe features - for example it allows interchange of HASH references and integers without any warning even with `use strict;` - something that even C doesn't allow. Also need for using all this complex references for nested arrays/hashes seems very unclear for me as mostly PHP programmer. I couldn't use PHP for this project because it seems PHP lacks any support for threads at all. Now I see threads in other languages are also experimental.
Finally, I know that the best ways for this job would be to use Async I/O however this is much more complex solution. So, I am basically in search for technology that has something like WWW::Mechanize and good production ready support for threading.