Perl Tail ?

perlAffen has asked for the wisdom of the Perl Monks concerning the following question:

Hi, In windows I am using a nice piece of perl code to tail a log file. It does a nice job until the file becomes very large, ie over 20 meg, then the perl process starts to be a memory hog. I tried using a cygwin tail, but it locks the file so the original process can't write to it. This perl code does NOT lock the file. Can someone suggest how I might grab new chunks of a large file efficiently ? The current code is like this

while(-s $file ) {
  open(TF,$file) || last ;
  seek(TF,$curpos,0);
  @lines=<TF>;
  $cursorposition = tell(TF);
  last if ((stat(_))[7] < $cursorposition);
    foreach $nline (@lines){
    $newline = substr $nline, 24;
    # write out to tailfile    $newline
    }    
  close(TF);
  sleep 1;
}
}
else
    { sleep 5; }
}
[download]

Thanks

Comment on Perl Tail ? Download Code

Replies are listed 'Best First'.
Re: Perl Tail ? by marto (Cardinal) on Oct 04, 2005 at 15:34 UTC
Hi, Further to the above there is also File::Tail. Hope this helps. Martin	[reply]
Re: Perl Tail ? by ikegami (Patriarch) on Oct 04, 2005 at 15:31 UTC
`tail` can efficiently be done using File::ReadBackwards. `# Displays the last $n lines of $file_name, # earliest first. my $file_name = 'file.log'; my $n = 5; my $fh = File::ReadBackwards->new($file_name) or die("Can't read input file: $!\n"); my @lines; for (1..$n) { my $line = $fh->readline(); last if not defined $line; unshift(@lines, $line); } print foreach @lines;` [download] Update: marto's suggestion seems to be more appropriate.	[reply] [d/l] [select]
Re: Perl Tail ? by rinceWind (Monsignor) on Oct 04, 2005 at 15:36 UTC
A super search on tail will give examples of how to do this, including my post Re: Log file tailing. -- Oh Lord, won’t you burn me a Knoppix CD ? My friends all rate Windows, I must disagree. Your powers of persuasion will set them all free, So oh Lord, won’t you burn me a Knoppix CD ? (Missquoting Janis Joplin)	[reply]
Re: Perl Tail ? by LanceDeeply (Chaplain) on Oct 04, 2005 at 15:50 UTC
i tried Perl Power Tools v0.14: tail and it seems to work ok in windows. i havent tried it on a 20 meg file though... hth	[reply]
Re: Perl Tail ? by Skeeve (Parson) on Oct 04, 2005 at 15:32 UTC
Is there any need for you to read in the whole file? `@lines=<TF>;` [download] Why not simply read line by line and don't store them? `$\=~s;s.;q^\|D9JYJ^^qq^\//\\\///^;ex;print`	[reply] [d/l] [select]
Re^2: Perl Tail ? by perlAffen (Sexton) on Oct 04, 2005 at 15:46 UTC
So this is an efficient way to do the Unix equivalent ? `tail -f logfile > tailfile` [download] I would like to tail it for 10 minutes, restart the tail process for another 10 mintues, copying off tailfile to tailfile.chunk before writing to tailfile again. COuld I use this to do it ? I will check tail archives too.	[reply] [d/l]
Re^3: Perl Tail ? by QM (Parson) on Oct 04, 2005 at 18:04 UTC
I recall many years ago that unix tail is broken on large files, such that I wrote my own Perl tail for one job. You might want to look at tail for Perl Power Tools. I would like to tail it for 10 minutes, restart the tail process for another 10 mintues, copying off tailfile to tailfile.chunk before writing to tailfile again Please be more specific. Are you intending to skip (lose) 10 minutes of logfile? Do you just want to split the logfile into chunks at 10 minute intervals? What is the fundamental problem you're trying to solve? -QM -- Quantum Mechanics: The dreams stuff is made of	[reply]
Re^4: Perl Tail ? by perlAffen (Sexton) on Oct 05, 2005 at 03:10 UTC
Re^4: Perl Tail ? by Anonymous Monk on Oct 04, 2005 at 20:28 UTC
Re: Perl Tail ? by graff (Chancellor) on Oct 04, 2005 at 22:57 UTC
I tried using a cygwin tail, but it locks the file so the original process can't write to it. LOL -- that's the best joke I've seen about MS Windows in a while. I suppose it could just be taken as a (yet another?) case of midguided implementation on the part of cygwin, but based on the whole brain-damaged approach to file locking foisted by MS, cygwin should not take all the blame (or should at least admit only to justifiable failure).	[reply]