I tried this (using open and curl) and performance took too big a hit. The cpu was doing about 15% user and 85% system and I could only get about 40 req/s - I need to be around 70 to 100, which I am able to do when running many disconnected non-threaded scripts.

Starting up a process for each request unfortunately is too expensive, thank you for the suggestion, though.

Starting up a process for each request unfortunately is too expensive

Then I would write one script that takes URLS from stdin and does the fetches.

#! perl -slw
use strict;
use LWP::Simple;

while( <> ) {
    chomp;
    my $content = get $_;
    ### do something with it.
}
[download]

And then drive multiple copies of that script from a threaded script:

#! perl -slw
use strict;
use threads;
use Thread::Queue;

sub worker {
    my $Q = shift;

    open my $pipe, '-|', q[ perl theOtherScript.pl] or die $!;
    while( my $url = $Q->dequeue() ) {
        print $pipe $url;
    }    
}

our $THREADS //= 4;
my $Q = new Thread::Queue;

my @workers = map threads->create( \&worker, $Q ), 1 .. $THREADS;
$Q->enqueue( $url ) while ...; ## fetch urls from somewhere and Q them
$Q->enqueue( (undef) x $THREADS );
$_->join for @workers;
[download]

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Great suggestion, thank you. I've implemented this and it works beautifully.

I chose to use IPC::Open2 to get bi-di communication, sending single-line messages back and forth, so the script is now as functional as it would have been with a thread-safe Crypt::SSLeay.