Hi Monks,

I have been trying to speed up a cpu-bound log file analysis program using threads. However the resulting program is even slower (5 times!) than my single threaded version. I have isolated the problem down to the following simple example. I must be doing something wrong. Any suggestions on how to get back to the sub second file reads?

For ref source files are about 4MB- 10MB with line lengths around 250bytes. I have only tested this on perl 5.10.1 in cygwin. 

 
#!/usr/bin/perl
# - Example threading code reading from multple files being slow 
use strict;
use warnings;
use threads;
use Thread::Queue;

my @files = qw( 
    SyslogR_20110615_134243.txt SyslogR_20110615_162146.txt 
    SyslogR_20110620_090237.txt SyslogR_20110620_090308.txt 
    SyslogR_20110620_092240.txt SyslogR_20110620_092328.txt
);


non_thread_process( \@files);
my $PARALLEL = 1;
thread_process( \@files);
$PARALLEL = 4;
thread_process( \@files);

sub non_thread_process {
    my $args=shift;
    for my $file (@$args) {
        process($file, "Non Thread");    
    }
}

sub thread_process {
    my $args = shift;
    my @threads;
    my $queue = Thread::Queue->new();
    for my $p ( 1 .. $PARALLEL ) { 
        push @threads, threads->create( sub {
            while( my $file = $queue->dequeue) { 
                process($file, "Thread $p of $PARALLEL");
        }})
    }
    $queue->enqueue( @$args, (undef) x $PARALLEL );
    $_->join for @threads;
    return 1;
}

sub process {
    my ($file, $msg) = @_;
    my $start =time;
    open my $IN, $file or die "Cannot read file $file : $!";
    while(<$IN>) {
        my @x= split/ /; # Do something
    }
    close $IN;    
    my $end =time;
    printf "$msg: Process of $file complete in %ds\n", $end-$start;
}
__END__
$ perl -v

This is perl, v5.10.1 (*) built for i686-cygwin-thread-multi-64int
(with 13 registered patches, see perl -V for more detail)

Copyright 1987-2009, Larry Wall

Perl may be copied only under the terms of either the Artistic License
+ or the
GNU General Public License, which may be found in the Perl 5 source ki
+t.

Complete documentation for Perl, including FAQ lists, should be found 
+on
this system using "man perl" or "perldoc perl".  If you have access to
+ the
Internet, point your browser at http://www.perl.org/, the Perl Home Pa
+ge.

$ perl ./example.pl


Non Thread: Process of SyslogR_20110615_134243.txt complete in 0s
Non Thread: Process of SyslogR_20110615_162146.txt complete in 1s
Non Thread: Process of SyslogR_20110620_090237.txt complete in 0s
Non Thread: Process of SyslogR_20110620_090308.txt complete in 1s
Non Thread: Process of SyslogR_20110620_092240.txt complete in 0s
Non Thread: Process of SyslogR_20110620_092328.txt complete in 1s
Thread 1 of 1: Process of SyslogR_20110615_134243.txt complete in 0s
Thread 1 of 1: Process of SyslogR_20110615_162146.txt complete in 1s
Thread 1 of 1: Process of SyslogR_20110620_090237.txt complete in 0s
Thread 1 of 1: Process of SyslogR_20110620_090308.txt complete in 0s
Thread 1 of 1: Process of SyslogR_20110620_092240.txt complete in 1s
Thread 1 of 1: Process of SyslogR_20110620_092328.txt complete in 0s
Thread 4 of 4: Process of SyslogR_20110620_090308.txt complete in 11s
Thread 1 of 4: Process of SyslogR_20110615_134243.txt complete in 16s
Thread 3 of 4: Process of SyslogR_20110620_090237.txt complete in 24s
Thread 2 of 4: Process of SyslogR_20110615_162146.txt complete in 29s
Thread 1 of 4: Process of SyslogR_20110620_092328.txt complete in 14s
Thread 4 of 4: Process of SyslogR_20110620_092240.txt complete in 19s

[download]

In reply to Reading from file in threaded code is slow by amcglinchy

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.