in reply to Parse Log File As Written

One possible solution can be using Tie::File.

"Tie::File represents a regular text file as a Perl array. Each element in the array corresponds to a record in the file. The first line of the file is element 0 of the array; the second line is element 1, and so on. The file is not loaded into memory, so this will work even for gigantic files. Changes to the array are reflected in the file immediately."

You may write a program which runs every few ms or reside in the memory. One sample program can be 

#/usr/bin/perl

$file = "my-log_file.log";

$o = tie @array, 'Tie::File', $file;

$o->flock; # lock the file. See Note 1.
$line_count = 0;

while( 1 ){
  
  if( $line_count < $#array ){ #New line is added
    $line_count++;
    foreach $line($line_count .. $#array )
    {

    #### Your code to parse @array[$line];####

    }
    $line_count = $#array;  
  }
  sleep(1); #parse every second
}

[download]
Hope this helps.

Notes :

  1. Tie::File docs say, "All the usual warnings about file locking apply here. In particular, note that file locking in Perl is advisory, which means that holding a lock will not prevent anyone else from reading, writing, or erasing the file; it only prevents them from getting another lock at the same time. Locks are analogous to green traffic lights: If you have a green light, that does not prevent the idiot coming the other way from plowing into you sideways; it merely guarantees to you that the idiot does not also have a green light at the same time."
  2. Since Tie::File won't load the file into memory, it avoids brute force read of entire file.
  3. This is just a possible method and the code is NOT tested.

Updates :

Cheers !

--VC


There are three sides to any argument.....
your side, my side and the right side.

Replies are listed 'Best First'.
Re^2: Parse Log File As Written
by dsheroh (Monsignor) on Sep 24, 2007 at 14:44 UTC
    "Changes to the array are reflected in the file immediately."

    Note that the Tie::File docs do not say here (or anywhere else) that changes to the file will be reflected in the array immediately. Tie::File may be capable of following additions to a ever-growing file, but that really doesn't appear to be one of the tasks it was designed to handle.

    A couple more relevant bits from the locking section of the Tie::File docs are

    When you use flock to lock the file, Tie::File assumes that the read cache is no longer trustworthy, because another process might have modified the file since the last time it was read. Therefore, a successful call to flock discards the contents of the read cache and the internal record offset table.

    and

    The best way to unlock a file is to discard the object and untie the array.

    To me, these indicate that, if the underlying file is changed, then Tie::File will have to rebuild its table of record offsets, which means starting over from the beginning of the file and reading every line until it reaches the one you're looking for (i.e., the one that was just added to the end of the file), which is exactly the sort of re-scan we're trying to avoid here.

[reply]

      "Changes to the array are reflected in the file immediately."

      This is something I copied from CPAN's Tie::File. it IS the 3rd line of the description.

      The CPAN doc gives other caveats of Tie::File

      Cheers !

      --VC


      There are three sides to any argument.....
      your side, my side and the right side.

[reply]
[d/l]
[reply]
        Yes, I saw it there. I didn't dispute that the docs say the array immediately updates the file. However, they do not say that the file immediately updates the array. Array -> file does not imply file -> array.
[reply]

In Section Seekers of Perl Wisdom