Re: Reading from file in threaded code is slow

update...

Ooops tried to retract post after discovering that I had mis-read what was happening but wasn't fast enough on the trigger.... Sorry I lost original wrong comment...but it was wrong anyway.

Update Again: here was original post:

---
I think that there is a flaw in your benchmark. Many file system resources are allocated on a per process basis, not per thread. It looks to me like your threaded version has a single process that's trying to open the same file 4 times and read from 4 different file positions? This can potentially cause big trouble and fighting.

I suggest recoding the benchmark so that no file is open more than once in the same process. In this case, that means each thread needs to be doing an independent list of files.

---
I was mistaken -- the OP's code already does what I suggested (the overall process never has the same file open twice - each thread is independent) - see BrowserUk's response..

Comment on Re: Reading from file in threaded code is slow

Replies are listed 'Best First'.
Re^2: Reading from file in threaded code is slow by BrowserUk (Patriarch) on Jul 08, 2011 at 20:47 UTC
It looks to me like your threaded version has a single process that's trying to open the same file 4 times and read from 4 different file positions? In the routine `thread_process()`, the list of files is pushed onto a queue. Each of the threads then pull (different) filenames from that queue: `sub thread_process { my $args = shift; my @threads; my $queue = Thread::Queue->new(); for my $p ( 1 .. $PARALLEL ) { push @threads, threads->create( sub { while( my $file = $queue->dequeue) { process($file, "Thread $p of $PARALLEL"); }}) } $queue->enqueue( @$args, (undef) x $PARALLEL ); $_->join for @threads; return 1; }` [download] So no. They are not reading from the same file. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^2: Reading from file in threaded code is slow
by BrowserUk (Patriarch) on Jul 08, 2011 at 20:47 UTC

It looks to me like your threaded version has a single process that's trying to open the same file 4 times and read from 4 different file positions?

In the routine thread_process(), the list of files is pushed onto a queue. Each of the threads then pull (different) filenames from that queue:

sub thread_process {
    my $args = shift;
    my @threads;
    my $queue = Thread::Queue->new();
    for my $p ( 1 .. $PARALLEL ) { 
        push @threads, threads->create( sub {
            while( my $file = $queue->dequeue) { 
                process($file, "Thread $p of $PARALLEL");
        }})
    }
    $queue->enqueue( @$args, (undef) x $PARALLEL );
    $_->join for @threads;
    return 1;
}
[download]

So no. They are not reading from the same file.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

[reply]
[d/l]
[select]