This uses threads to achieve a near perfect 75% reduction on overall runtime using 4 threads:
#! perl -slw
use strict;
use Time::HiRes qw[ time ];
use threads;
use threads::shared;
my @buffers;
sub thread {
my $tid = threads->tid;
print "[$tid] starting";
my $bufRef = shift;
open my $fh, '<', $bufRef;
while( <$fh> ) {
## next if ... ## add pre-filter criteria here is applicable
my @f = split "\t", $_;
## do something with the data
}
print "[$tid] ending";
}
our $N //= 199;
our $T //= 4;
my $start = time;
my @threads;
for( 0 .. $N-1 ) { ## glob '*.tsv or whatever
my $bufRef = \$buffers[ $_ % $T ];
{
local( @ARGV, $/ ) = 'numbers.tsv';
$$bufRef = <>;
}
push @threads, async( \&thread, $bufRef );
while( @threads >= $T ) {
sleep 1 until threads->list( threads::joinable );
for( reverse 0 .. $#threads ) {
next unless $threads[ $_ ]->is_joinable;
$threads[ $_ ]->join;
splice @threads, $_, 1;
}
}
}
$_->join for @threads;
printf "Took %f seconds\n", time() - $start;
__END__
C:\test>1036737-3 -T=1 -N=8
[1] starting
[1] ending
[2] starting
[2] ending
[3] starting
[3] ending
[4] starting
[4] ending
[5] starting
[5] ending
[6] starting
[6] ending
[7] starting
[7] ending
[8] starting
[8] ending
Took 160.641885 seconds
C:\test>1036737-3 -T=4 -N=8
[1] starting
[2] starting
[3] starting
[4] starting
[2] ending
[1] ending
[5] starting
[4] ending
[3] ending
[6] starting
[7] starting
[8] starting
[5] ending
[6] ending
[7] ending
[8] ending
Took 39.710515 seconds
With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.