Re: Optimising a search of several thousand files

Try running this:

#!/usr/bin/perl -wl
use strict;

my $dir = qw(/home/idlerpg/graphdump);

opendir( DIR, $dir ) or die "Cannot open $dir:$!";

my @files     = reverse sort readdir(DIR);
my $currlevel = 68;
my $numfiles;
my $numlines;
my $totalfiles = scalar @files;
my $user;
my $level;

for my $file (@files) {

    open IN, '<', "$dir/$file" or die "Ack!: $!";
    $numfiles++;

    while (<IN>) {

        $user  = '';
        $level = '';
        $numlines++;
        chomp();
        ( $user, $level ) = ( split /\t/o )[ 0, 3 ];
        next if $user ne 'McDarren';
        last if $level <= $currlevel;
        print $file, $user, $level;

    }

    close(IN);
}

print "Processed $numlines lines in $numfiles files (total files: $tot
+alfiles)";
[download]

Of course, I didn't test it, so I'm not sure it will work as you expect.

Alceu Rodrigues de Freitas Junior
---------------------------------
"You have enemies? Good. That means you've stood up for something, sometime in your life." - Sir Winston Churchill

Comment on Re: Optimising a search of several thousand files Download Code

Replies are listed 'Best First'.
Re^2: Optimising a search of several thousand files by McDarren (Abbot) on Jan 30, 2007 at 05:08 UTC
As far as I can see, the only real change you made there from my original code was to add the /o modifier to the pattern match within the split. This doesn't seem to make any noticeable difference.	[reply]
Re^3: Optimising a search of several thousand files by glasswalk3r (Friar) on Jan 31, 2007 at 12:06 UTC
Did you tested it against Benchmark? There are some perl monks that believe that `qr` works better then using `/o` in regex matching (but I never tried to compare both). Anyway, if you use one of them you might get better results since the regex will not be compiled again and again. Updated: sorry, `/o` will do absolute nothing as described below: It should be immediately apparent that using /o on a regex with no interpolation has zero effect. You could use it on a regex that uses interpolation but then you get delayed runtime effects. In general I'd much prefer all of my compilation to occur at compilation time. As a matter of practice string evals are restricted, this works similarly. If you use qr// expressions then you've shunted the regex compilation and syntax checking off to the normal script compilation time. The alternative is that you discover a syntax error at runtime. Yuck. More information about that is here: /o is dead, long live qr//!. By the other hand, `qr` should work. Other suggestion is that you try to play around with `unpack` and see if you cannot parse the line without regular expression. The other things I see you could try is to load the entire file into memory and use `foreach` instead using `while` (of course, this will depend of the file size and available memory that you have). Also you may try to instead of `print $file, $user, $level` put this information into an array to print at the end of the script, to avoid system calls. And as last resources (since these may give you a lot of work but bringing little results), you may want to remove `use strict` and `-w`, as you're not gonna need them if you consider that the code is stable enough. `perlcc -B` may also help, specially because your program is not importing any module. One thing that I didn't understand in your code is why it doesn't close the filehandles. The block labels were also not necessary there and the last `print` statement could be used outside the loop. Alceu Rodrigues de Freitas Junior --------------------------------- "You have enemies? Good. That means you've stood up for something, sometime in your life." - Sir Winston Churchill	[reply] [d/l] [select]