in reply to Re^2: Perl && threads && queues - how to make all this work together
in thread Perl && threads && queues - how to make all this work together
You code is so full of
eg. $pos is a shared variable accessed and modified by every thread, but you never lock it.
(I used binary access to be able to operate very large files)
You do not need to use binary access in order to handle huge files in Perl (nor any other language I'm aware of).
Which makes all the laborious (and mostly broken) effort you went to to write your own line handling:
Why abs($threadscnt-1)?
it is doubtful that it would ever work reliably.
You don't say why you wish to avoid using Thread::Queue, but your understanding of Perl, much less your understanding of threading, isn't sufficient to allow you to consider writing your own shared data handling.
By way of encouragement, this code does pretty much exactly what your code attempts to do:
#! perl -slw use strict; use threads qw[ yield ]; use threads::shared; use Thread::Queue; use Time::HiRes qw[ sleep ]; use constant NTHREADS => 30; my $pos :shared = 0; open FILE, '<', $ARGV[ 0 ] or die $!; my $size = -s FILE; sub thread { my $Q = shift; my $tid = threads->tid; while( my $line = $Q->dequeue ) { printf "%3d: (%10d, %10d) :%s", $tid, $pos, $size, $line; sleep rand 5; } } my $Q = Thread::Queue->new; my @threads = map threads->create( \&thread, $Q ), 1 .. NTHREADS; while( !eof FILE ) { sleep 0.001 while $Q->pending; for( 1 .. NTHREADS ) { $Q->enqueue( scalar <FILE> ); lock $pos; $pos = tell FILE; } } $Q->enqueue( (undef) x NTHREADS ); $_->join for @threads;
It's clear, clean and simple. And works. (Though it is of dubious value, but you wrote the spec!)
If the idea of threading your code is to allow you to process your huge file more quickly, that probably isn't going to work unless you spend an inordinate amount of time processing each line. And if that's the case, unless you're using hardware with 16 or more cores, using 30 threads is unlikely to be an optimum strategy.
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^4: Perl && threads && queues - how to make all this work together
by xaero123 (Novice) on Feb 12, 2010 at 17:13 UTC | |
by BrowserUk (Patriarch) on Feb 12, 2010 at 18:08 UTC | |
|
Re^4: Perl && threads && queues - how to make all this work together
by xaero123 (Novice) on Feb 13, 2010 at 18:55 UTC | |
by BrowserUk (Patriarch) on Feb 13, 2010 at 19:39 UTC |