in reply to Optimising a search of several thousand files

I'd suggest using a binary search on @files to determine when your level change occurred, rather than reading backwards through time by 5 minute increments.

Alternatively, you could start by jumping backwards in time by 1 week intervals, then when you find the level change you could move forward in 1 day intervals, then backwards in 1 hour intervals, then forward in 5 minute intervals.

You could combine a date filter with either approach. Since you know the bot died around Christmas, it seems silly to test the approximately 8000 files between today and Jan 1, 2007 (or those created prior to about Dec 15, 2006). If you narrow your search window to a week or two, I'd be surprised if any method took very long. :-)

Replies are listed 'Best First'.
Re^2: Optimising a search of several thousand files
by McDarren (Abbot) on Jan 29, 2007 at 05:59 UTC
    heh, yeah....

    Simply by skipping any files that are less than 20 days old, I get an answer within 5 seconds :) 

    $ time ./bobf.pl
dump.1167332700 McDarren 71
Processed 367155 lines in 3298 files (total files:57872)
        4.41 real         4.20 user         0.14 sys

    [download]
[reply]
[d/l]

In Section Seekers of Perl Wisdom