in reply to Re^4: Threads and HTTPS Segfault
in thread Threads and HTTPS Segfault
Starting up a process for each request unfortunately is too expensive
Then I would write one script that takes URLS from stdin and does the fetches.
#! perl -slw use strict; use LWP::Simple; while( <> ) { chomp; my $content = get $_; ### do something with it. }
And then drive multiple copies of that script from a threaded script:
#! perl -slw use strict; use threads; use Thread::Queue; sub worker { my $Q = shift; open my $pipe, '-|', q[ perl theOtherScript.pl] or die $!; while( my $url = $Q->dequeue() ) { print $pipe $url; } } our $THREADS //= 4; my $Q = new Thread::Queue; my @workers = map threads->create( \&worker, $Q ), 1 .. $THREADS; $Q->enqueue( $url ) while ...; ## fetch urls from somewhere and Q them $Q->enqueue( (undef) x $THREADS ); $_->join for @workers;
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^6: Threads and HTTPS Segfault
by onelesd (Pilgrim) on Aug 23, 2011 at 05:51 UTC |