in reply to Re^2: Perl && threads && queues - how to make all this work together
in thread Perl && threads && queues - how to make all this work together

You code is so full of

it is doubtful that it would ever work reliably.

You don't say why you wish to avoid using Thread::Queue, but your understanding of Perl, much less your understanding of threading, isn't sufficient to allow you to consider writing your own shared data handling.

By way of encouragement, this code does pretty much exactly what your code attempts to do:

#! perl -slw use strict; use threads qw[ yield ]; use threads::shared; use Thread::Queue; use Time::HiRes qw[ sleep ]; use constant NTHREADS => 30; my $pos :shared = 0; open FILE, '<', $ARGV[ 0 ] or die $!; my $size = -s FILE; sub thread { my $Q = shift; my $tid = threads->tid; while( my $line = $Q->dequeue ) { printf "%3d: (%10d, %10d) :%s", $tid, $pos, $size, $line; sleep rand 5; } } my $Q = Thread::Queue->new; my @threads = map threads->create( \&thread, $Q ), 1 .. NTHREADS; while( !eof FILE ) { sleep 0.001 while $Q->pending; for( 1 .. NTHREADS ) { $Q->enqueue( scalar <FILE> ); lock $pos; $pos = tell FILE; } } $Q->enqueue( (undef) x NTHREADS ); $_->join for @threads;

It's clear, clean and simple. And works. (Though it is of dubious value, but you wrote the spec!)

If the idea of threading your code is to allow you to process your huge file more quickly, that probably isn't going to work unless you spend an inordinate amount of time processing each line. And if that's the case, unless you're using hardware with 16 or more cores, using 30 threads is unlikely to be an optimum strategy.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"I'd rather go naked than blow up my ass"

Replies are listed 'Best First'.
Re^4: Perl && threads && queues - how to make all this work together
by xaero123 (Novice) on Feb 12, 2010 at 17:13 UTC
    OK, thank you very much jethro and BrowserUk! But I have the last question. BrowserUk, why did you use "threads qw yield ;" and never used the yield() function? What effect is this have?
      BrowserUk, why did you use "threads qw yield ;" and never used the yield() function?

      I simply forgot to remove the reference. I had originally coded sleep 0.001 while $Q->pending;

      As yield while $Q->pending;, but switched it because yield() can render to a very tight loop, which consumes large amount of cpu needlessly.

      yield equates to sleep 0 which basically relinquishes the rest of the current timeslice. But, if no other thread (or process) is ready to run, it returns very quickly making for a cpu-sapping tight loop.

      On my system, the above code with yield(), consumes 25% cpu. Ie. 100% of one core.

      The same code with sleep 0.001 consumes so little cpu it gets measured as 0.00%, but it mokes no difference to the overall runtime of the program.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
Re^4: Perl && threads && queues - how to make all this work together
by xaero123 (Novice) on Feb 13, 2010 at 18:55 UTC
    Ok, thanks, BrowserUk! It is impossible to underestimate your help! :))
      impossible to underestimate your help!

      Is that like saying "No help at all!"? :)