use Time::Piece;

my @vkeys;
my %ranges_dt;

# ==================================================
sub check_date_in_range
{
    my ($value, %h) = @_;
    my $cvalue = Time::Piece->strptime($value, '%m/%d/%Y-%H:%M:%S')->e
+poch;
    if (!@vkeys) {
        @vkeys = sort keys %h;
    }
    foreach my $k (@vkeys) {
        if (($cvalue >= $h{$k}{start}) && ($cvalue <= $h{$k}{end})) {
            return $k;
        }
    }
    return;
}

while (<DATA>) {
    chomp;
    if (s/^(\w+)://) {
        my $cat = $1;
        if ($cat eq "R") {
            my ($i, $s, $e) = split ',';
            $ranges_dt{$i}{start} = Time::Piece->strptime($s, '%Y-%m-%
+d %H:%M:%S')->epoch;
            $ranges_dt{$i}{end} = Time::Piece->strptime($e, '%Y-%m-%d 
+%H:%M:%S')->epoch;
        } else {
            my $rangeid = check_date_in_range($_, %ranges_dt);
            if (!defined $rangeid) {
                print "No range found for $_\n";
            } else {
                print "Found range: $rangeid\n";
            }
        }
    }
}
[download]

Another option would be not to store the timestamps in a hash and sort them every time a new one arrives, but use a data structure that can add a new element and quickly search for the closest members (binary search tree, for example).

($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord
}map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,
[download]

Boy, there is a huge difference in performance between Date::Manip::Date and Time::Piece ...

[user@host newload]$ ./t-benchdates.pl 500
Benchmark: timing 500 iterations of Date::Manip::Date, Time::Piece...
Date::Manip::Date: 124 wallclock secs (119.35 usr +  1.30 sys = 120.65
+ CPU) @  4.14/s (n=500)
Time::Piece:  5 wallclock secs ( 3.88 usr +  0.03 sys =  3.91 CPU) @ 1
+27.88/s (n=500)
[download]

Certainly a code change I will be making but that leaves the foreach loop through all available ranges as a massive performance hit still ...

I assume your recommendation for Time::Piece is due to better performance?

Otherwise, not sure why I would change libraries from something I know works.

As to sorting the keys each time, I already reduced that workload to a cached array with the code:

    if (!@vkeys) {
        @vkeys = sort keys %h;
    }
[download]

Only one sort is being done the first time the sub is run ...

Alternatively, why not just store the line number of the last line processed between runs? It will be much faster to just discard the first N lines on the next run than to do date-based processing on them.

That is a good question.

The reason is that the logs are uploaded to a central server for processing and are contained in a .tgz file. Once opened up, there is no guarantee that each hour being processed by the loader script is in sequential order and the processing of the log files is done in parallel in a queue with work handed out to child processing scripts. So, the DB is the only point of reference. Also, storing a byte count would not work when the files are rotated (supposedly once a day) and the file starts over at 0 (but on some systems the logrotate does not always happen so those files just keep growing).

It could be that hour 3, 5 and 6 are processed and THEN hour 4 gets processed so again the byte count from hour 6 will not be applicable to hour 4. But with hour 4 having been included in the hour 5 and hour 6 upload, there is no need to actually pull out and load the data but without checking the dates for inclusion in the range, there is no way to guarantee this.

Given all of this, dates are the only reliable means of knowing data has been processed before.

But, even with being able to process only dates after a byte skip, I would still need to process the dates in the range. And there are a lot of them each hour on some of the busier systems.

Hence, the original question, how to make the lookup faster/more efficient?

Can the ranges overlap each other ? In the 3 shown as example they don't.


poj

Also, I can't really believe that this I/O bound procedure would in any way be impacted by the number of nanoseconds required to manipulate the dates.

Never underestimate the impact of a large number of subroutine calls.

Given the OP's followup, it seems there's a lot going on in the original code.

-QM
--
Quantum Mechanics: The dreams stuff is made of

Yes, there is a lot going on. Details on the problem and discussion on solutions are in an earlier post from me:

http://www.perlmonks.org/?node_id=1207429

But, introducing the has range lookup was Terrible for the performance as it meant I had to call a sub, search through about 500 hash items, return if found; AND it had to do this for between 250,000 and 500,000 items each run. If you look at my stats output, you will see this was taking about (edited numbers to reflect actual data) an average of 120 seconds per 1000.

Changing to the array lookup was a HUGE increase in speed knocking the per 1000 time to 5-7 seconds.

With all of the other stuff going on, it is a performance nightmare ... Every little bit counts.

Is %ranges_dt (reference data from DB) once loaded unchanged throughout processing log data? It doesn't seem to be in the code.