in reply to Optimising a search of several thousand files

For each file, set $/ to "\nMcDarren\t", read a line, set $/ back to "\n", read another line, and decide if this is the right file?

Though I'd probably actually do something like: ls -t|xargs egrep '^McDarren.*71'|head -1

I'm looking for a job.

Replies are listed 'Best First'.
Re^2: Optimising a search of several thousand files
by McDarren (Abbot) on Jan 30, 2007 at 05:33 UTC
    I gave that a go. 
    FILE:
for my $file (@files) {
    my ($stamp) = $file =~ /dump.(.*)/;
    next FILE if $stamp > ($now - $start);
    $numfiles++;
    $/ = "\nMcDarren\t";
    open IN, '<', "$dir/$file" or die "Ack!:$!";
    my $data = <IN>;
    $/ = "\n";
    $data = <IN>;
    my ($level) = (split /\t/,$data)[2];
    next FILE if $level <= $currlevel;
    print "$file - $level";
    print "Processed $numfiles files (total files:$totalfiles)";
    exit;
}

    [download]
    Gives: 
    $ time perl ysth.pl
dump.1167332700 - 71
Processed 8764 files (total files:58154)
       39.55 real         2.35 user         0.80 sys

    [download]
    Note that I needed to make it skip the most recent 2 days of files as the game was restored yesterday.

    Regards the xargs/egrep solution - that also works (with a bit of minor tweaking), but isn't any faster than any of the Perl solutions.

    Thanks,
    Darren :)

[reply]
[d/l]
[select]
      Regards the xargs/egrep solution - that also works (with a bit of minor tweaking), but isn't any faster than any of the Perl solutions.
      Except in coding time :)

      I'm looking for a job.

[reply]
Re^2: Optimising a search of several thousand files
by diotalevi (Canon) on Jan 29, 2007 at 17:43 UTC

    That's clever but now you've committed to having everything prior to \nMcDarren\t in RAM. Perhaps that's an issue?

    ⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊

[reply]
[d/l]

In Section Seekers of Perl Wisdom