perlAffen has asked for the wisdom of the Perl Monks concerning the following question:

I have been trying to use this module to do a tail -f on a windows logfile. I had some perl code to do it that I posted before and below, but it was reading the whole file and thus using too much memory. I tried ppm to load File::Tail and it looks for File::HiRes, yet the PM goes in Time::HiRes. I have struggled for hours trying to get this PM loaded.
I am ready to pay someone to help me.
I just need an efficient way to that won't use alot of memory to 'tail -f' a large windows file. That is all.

perl somescript.pl thefile > outputfile

I have this 
$firstrun = 1;
while(1) {
    if ( -s $file ) {
    sleep 1;
open(TF,$file) || next ;
seek(TF,0,0);
@lines=<TF>;
my $curpos=tell(TF);
close(TF);
    if ( $firstrun < 1 ){
    foreach $lyne (@lines){
        printf $lyne;    
    }
    }
sleep 1;
$firstrun = 0;
while(-s $file ) {
  open(TF,$file) || last ;
  seek(TF,$curpos,0);
  @lines=<TF>;
  $curpos = tell(TF);
  last if ((stat(_))[7] < $curpos);
    foreach $lyne (@lines){
    printf $lyne;    
    }    
  close(TF);
  sleep 1;
}
}
else
    { sleep 5; }
}

[download]

but it is a memory hog. Plus, sometimes I am not allowed to install PMs like File::Tail.
I like this offered by another Monk, but I am not skilled enough (see above) to turn it into efficient code. 
while (1) {
    stat the file
    if size of file changed since last time {
        open file
        seek to previous EOF
        print file contents until EOF
        save EOF position for next time
        close the file
    }

    sleep for x seconds
}

[download]
Can anyone help with making the above not read through the whole file, possibly becoming more memory efficient ?
Thanks

Replies are listed 'Best First'.
Re: File::Tail on win32
by GrandFather (Saint) on Nov 29, 2005 at 23:20 UTC

    THe following sample code should do it for you. You should omit the updateFile related code - that's just there to get the test file updated.

    use strict;
use warnings;

my $logFileName = 'test.txt';
my $lastPos = 0;
my $endTime = time + 60;

while (time < $endTime)
  {
  next if ! -e $logFileName;
  next if $lastPos >= -s $logFileName;
  
  open inFile, '<', $logFileName;
  seek inFile, $lastPos, 0;
  print while <inFile>;
  $lastPos = tell inFile;
  close inFile;
  }
  continue
  {
  sleep (5);
  updateFile ();
  }

sub updateFile
{
open outFile, '>>', $logFileName;
print outFile 'Line added to file at ' . (join ', ', localtime) . "\n"
+;
close outFile;
}

    [download]

    Printed:

    Line added to file at 30, 16, 12, 30, 10, 105, 3, 333, 1
Line added to file at 35, 16, 12, 30, 10, 105, 3, 333, 1
Line added to file at 40, 16, 12, 30, 10, 105, 3, 333, 1
Line added to file at 45, 16, 12, 30, 10, 105, 3, 333, 1
Line added to file at 50, 16, 12, 30, 10, 105, 3, 333, 1
Line added to file at 55, 16, 12, 30, 10, 105, 3, 333, 1
Line added to file at 0, 17, 12, 30, 10, 105, 3, 333, 1
Line added to file at 5, 17, 12, 30, 10, 105, 3, 333, 1
Line added to file at 10, 17, 12, 30, 10, 105, 3, 333, 1
Line added to file at 15, 17, 12, 30, 10, 105, 3, 333, 1
Line added to file at 20, 17, 12, 30, 10, 105, 3, 333, 1

    [download]
    DWIM is Perl's answer to Gödel
[reply]
[d/l]
[select]
      thanks, I'll hook it up to large file and try it.
[reply]
      The script behaves strangely when the source file is removed and recreated.
      like so, test.txt is created by doing echo "" > test.txt
      run -> perl this.pl
      other window (ow) echo one >> test.txt
      one
      (ow) echo two >> test.txt
      two
      (ow) del /F test.txt
      (ow) echo uno > test.txt
      (ow) echo due >> test.txt
      (ow) echo tre >> test.txt
      re
      (ow) echo quattro >> test.txt
      quattro

      So it seems to hold on to the last cursor position and won't yield any output until that last postion is surpassed ?
      Also when you first run it, it outputs the entire file, is it possible to set $lastpos to its current EOF line ? ie Can it go to the bottom and wait for the next newline ?
      Also if I wanted to hand the output to a program, rather than print, how would I change
      print while <inFile>; ??
      Thanks !
[reply]

        This modified version may be closer to what you want:

        use strict;
use warnings;

my $logFileName = 'test.txt';
my $highWater = -1; # Signal first time through

while (1)
  {
  next if ! -e $logFileName;
  if ($highWater >= -s $logFileName)
    {
    $highWater = -s $logFileName; # Reset high water mark
    next;
    }
 
  open inFile, '<', $logFileName;
  seek inFile, $highWater, 0;
  #my @lines; # Uncomment to collate lines
  while (<inFile>)
    {
    print $_; # Replace with alternate code to handle output.
    #push @lines, $_; #Uncomment to collate lines
    }
  
  $highWater = tell inFile;
  close inFile;
  
  # Use @lines as required here to deal with added lines in one hit. e
+g.:
  #print @lines; #Uncomment to print collated lines
  }
  continue
  {
  sleep (5);
  }

        [download]
        DWIM is Perl's answer to Gödel
[reply]
[d/l]

Back to Seekers of Perl Wisdom