my point was merely that threaded code is hard to get right ...

If you don't follow the rules, the simplest things can screw up:

perl -e "fork while fork"
[download]

And here's a bit of the output:

Don't blame Perl or threads because your C runtime libraries are paying lip service to the realities of concurrency.

If it doesn't take care of things, you'll have to do it.

The addition of my $old = select $fh; $|++; select $old; might sort out the interleaving problem.

And use threads stack_size => 4096; will substantially reduce the memory footprint.

But that's all irrelevant. Like saying a bendy-bus is 25x better than a family car because it can carry 100 people.

The 500 do-almost-nothing threads in your code

Not my code!

And what is the point of running 500 threads?

It takes ~40MB to run 4 threads. That's a whole 2% of the ram of the lowest spec commodity box you'll ever find for sale.

And with 4 threads, it takes:

c:\test>junk39 -THREADS=4 >nul
51.556654214859 at C:\test\junk39.pl line 55, <> line 1
[download]

51 seconds to pull 501 names.

With 8 threads:

c:\test>junk39 -THREADS=8 >nul
49.5222151279449 at C:\test\junk39.pl line 55, <> line 1.
[download]

49 seconds. So doubling the number of threads gained almost nothing. The pipe or the remote server is the limiting factor.

I'd love to compare like with like, but having installed the 131 file that make up the POE behemoth:

ppm> install 1
Downloading POE-1.352...done
Downloading POE-Test-Loops-1.351...done
Unpacking POE-1.352...done
Unpacking POE-Test-Loops-1.351...done
Generating HTML for POE-1.352...done
Generating HTML for POE-Test-Loops-1.351...done
Updating files in site area...done
 131 files installed
[download]

I was still missing stuff your script needed:

c:\test>mbethke.pl
Can't locate POE/Component/Client/HTTP.pm in @INC (@INC contains: c:/P
+erl64/site/lib c:/Perl64/lib .)
BEGIN failed--compilation aborted at (eval 33) line 1.
could not import qw(Component::Client::HTTP) at C:\test\mbethke.pl lin
+e 4
BEGIN failed--compilation aborted at C:\test\mbethke.pl line 4.
[download]

So, then I tried to download that, but one of its dozen or so dependancies was unavailable:

1: POE-Component-Client-HTTP
   a HTTP user-agent component
   Version: 0.945
   Released: 2012-03-10
   Author: Rocco Caputo <rcaputo@cpan.org>
   Provide: POE::Component::Client::HTTP version 0.945
   Provide: POE::Component::Client::HTTP::Request version 0.945
   Provide: POE::Component::Client::HTTP::RequestFactory version 0.945
   Provide: POE::Filter::HTTPChunk version 0.945
   Provide: POE::Filter::HTTPHead version 0.945
   Require: HTTP::Headers version 5.81 or better
   Require: HTTP::Request version 5.811 or better
   Require: HTTP::Request::Common version 5.811 or better
   Require: HTTP::Response version 5.813 or better
   Require: HTTP::Status version 5.811 or better
   Require: Net::HTTP::Methods version 5.812 or better
   Require: POE version 1.312 or better
   Require: POE::Component::Client::Keepalive version 0.269 or better
   Require: Socket::GetAddrInfo version 0.19 or better
   Require: Test::More version 0.96 or better
   Require: Test::POE::Server::TCP version 1.14 or better
   Require: URI version 1.37 or better
   Repo: ActiveState Package Repository
   CPAN: http://search.cpan.org/dist/POE-Component-Client-HTTP-0.945/
ppm> install 1
ppm install failed: Can't find any package that provides Socket::GetAd
+drInfo for POE-Component-Client-HTTP
Can't find any package that provides Socket::GetAddrInfo for POE-Compo
+nent-Resolver
[download]

It might take a bit less memory, but it certainly won't be quicker because the limitation is the pipe and/or remote server.

I guess I could try installing the 270 module mojo behemothe, but it never terminates:

c:\test>zwon >nul
Too late to run CHECK block at c:/Perl64/site/lib/EV.pm line 84, <> li
+ne 501.
Terminating on signal SIGINT(2)
[download]

One way uses 0.5% of my memory; only needs what came installed with my Perl installation; and works.

The other two require gobs of extra code and either don't run or never finish. The decision is an easy one for me.

All that's left for me to do is free up about 50 MB of space on my harddrive by throwing away all the crap installed to write this post:

ppm> uninstall POE
POE: uninstalled
ppm> uninstall mojolicious
Mojolicious: uninstalled
[download]

There, all done.