Using File::ReadBackwards with Compressed Files

Dru has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,

I have compressed log files that I want to give users the option to search from the bottom of the file back. I found File::ReadBackwards which works well for uncompressed log files, but I can't figure out how to use it with my compressed log files. I pefer not to uncompress them first. Can someone show me the best way to do this?

How I'm opening and processing compressed log files:

open (FILE, "zcat $filename |") or die && error($file);
[download]

The portion that reads the file starting from the bottom:

# Run this portion if user chooses bottom
 
if ($pos =~ /\bbottom\b|\bb\b/){
 
  my $bw = File::ReadBackwards->new( $file ) 
                or die "can't read $file: $!\n" ;
 
  while( defined( $log_line = $bw->readline ) ) {
 
       print $log_line ;
 
}
[download]

Thanks,
Dru

Comment on Using File::ReadBackwards with Compressed Files Select or Download Code

Replies are listed 'Best First'.
Re: Using File::ReadBackwards with Compressed Files by Roger (Parson) on Mar 01, 2004 at 14:34 UTC
I don't think you can start processing a compressed file from the middle, unless for some simple compression algorithm. Your log file is most likely compressed with compress or gzip that use the LZ77/LZSS sliding window compression algorithm or its variants. The compression algorithm works on building dictionary of words, or phrase book, of repetitive patterns in a look ahead window, and re-encode the matching phrases with a token in the dictionary. And the encoding window moves forward. So I don't it is possible to decode a LZ compressed file backwards. Unfortunately you will have to uncompress the entire file first.	[reply]
Re: Using File::ReadBackwards with Compressed Files by matija (Priest) on Mar 01, 2004 at 14:46 UTC
I'm afraid you can't do that, Dave. (Sorry, 2001 flashback :-) An uncompressed logfile will make sense wherever you start to read it, as long as it is at the beginning of a line. However, a compressed file needs to be read from the start. In particular, modern compression alghorhythms change their coding tables as they move through the data they compress. That means you can't make sense of what came later, unless you know what came before. Even if you found a binary of a zReadBackwardCat it would still have to decompress/read through the whole file in order to understand the end. Some of the latest compression methods (bzip2?) have resync points scattered through the file, and you might be able to use those to mitigate the worst of it. But I think it would be a lot of work, and it would only work for that particular compression method.	[reply]
Re: Using File::ReadBackwards with Compressed Files by esskar (Deacon) on Mar 01, 2004 at 14:37 UTC
have you tried it like this `my $filepipe = "zcat $filename \|"; my $bw = File::ReadBackwards->new($filepipe) or die "can't read $filep +ipe: $!\n";` [download] but i'm not sure about that... but anyway... the module probably has to have read in the complete file into memory or temporary store it somewhere completly, so it will probably simpler to decompress it in the first place and than reading it backwards.	[reply] [d/l]