in reply to print log file

I assume that your real problem is that you want to traverse a large log file fast? There are certainly some CPAN module which do this -- and someone will post about it and make this look primitive. :-)

You could try a binary search with seek() to look for the general area to start looking. Something like:

# Find a good place to start traversing a large
# file for sorted data.

return 0 if $filelen < 2000000;  # A few MB? Please... :-)
my $safe_offset = 0;
my $jump = 0.5;
my $step = 0.25;
for(1..8) {
  # See if 
  my($testoff) = int($filelen * $jump);
  seek(FILE, $testoff, 0);

  if ( do_test(*FILE, $threshold) ) {
    $safe_offset = $testoff; # Point to offset for 
    $jump = $jump + $step;
  } else {
    $jump = $jump - $step;
  }
  $step = $step / 2.0;
}

# Go to selected place:
seek(FILE, $safe_offset);
<FILE> if $safe_offset > 0;

return $safe_offset;

sub do_test {
  my(*FILE, $compare) = @_;
  # Has moved somewhere in file. Skip partial line of log:
  <FILE>;
  my $log = <FILE>;
  # You write this (you know the format). Return true if OK:
  return date_test($log, $compare);
}

[download]

Just an idea, ignore if my assumption about your problem was wrong. Code is untested since I'm busy. I'll be back in about a work day and can write more then, if you need more details.

I hope this won't embarrass me when I get back. (-: On the other hand -- I'll make someone's day when they get to point out errors. :-)

Update: Added return stuff, so it sets up offset to file correctly.

Replies are listed 'Best First'.
Re^2: print log file
by Fengor (Pilgrim) on Nov 30, 2006 at 07:20 UTC
    Since there is always more than one way to do it. Here would be my guess to traverse the file faster: 
    my $jmp_distance = 100;
my $treshhold = $FOO

my $line = <FILE>;
(my $tstamp, my $rest) = split(/\s+/,$line,2);

#skipping through file by jumping over $jmp_distance lines
do {
  $. += $jmp_distance;
  $line = <FILE>;
  ($tstamp, $rest) = split(/\s+/,$line,2);
} while ($tstamp<=$treshhold)

#since we overshot with the last jump
$. -= $jmp_distance;

while (<FILE>) {
  ($tstamp, $rest) = split(/\s+/,$_,2);
  print if $tstamp > $treshhold;
}

    [download]
    Of course you can modify this further. For example putting the whole do whileloop in a sub and recursively call it with shorter jmp_distances. that way you could start with a distance of 10000 and then go to shorter distances like 1000, 100, 10 and 1 (1 = condition to break the recursion) if you call the sub with jmp_distance/10.

    --
    "WHAT CAN THE HARVEST HOPE FOR IF NOT THE CARE OF THE REAPER MAN"
    -- Terry Pratchett, "Reaper Man"

[reply]
[d/l]
Re^2: print log file
by xiaoyafeng (Deacon) on Nov 30, 2006 at 09:43 UTC
    Wow,code of great wisdom!although it isn't for my problem. When will you be back? I can't wait to read more!
[reply]
      Hrm, I'll take that for humorous irony and not sarcasm. :-)
[reply]
Re^2: print log file
by Fengor (Pilgrim) on Nov 30, 2006 at 06:44 UTC
    hmm first: seek is byte oriented not line oriented. As far as i understood it he needs one line at a time. This isnt much of a problem when you have a file with fixed length lines but since we are talking about a logfile this is doubtful (although possible). So while seek is probably the fastest way to move around a file i'm afraid you cant use it here.

    --
    "WHAT CAN THE HARVEST HOPE FOR IF NOT THE CARE OF THE REAPER MAN"
    -- Terry Pratchett, "Reaper Man"

[reply]
      That is why the do_test routine skipped the first line it read. It said: 

        # Has moved somewhere in file. Skip partial line of log:
  <FILE>;

      [download]
      This just attempts to make the log traversal smaller, not perfect. (Wouldn't be so hard either, of course. But I'm out the door in a few minutes. :-)
[reply]
[d/l]
[select]
        ok i withdraw ;) still too early here should get more caffeine in the bloodstream before writing posts.

        --
        "WHAT CAN THE HARVEST HOPE FOR IF NOT THE CARE OF THE REAPER MAN"
        -- Terry Pratchett, "Reaper Man"

[reply]

In Section Seekers of Perl Wisdom