How would your approach work with a normal file?

If you show us the code you have already, then we can likely help you much better on how to approach moving from a normal file to a compressed file. Have you considered just decompressing your compressed file?

Personally, I'm fond of just using the pipe-open like this:

my $file = 'some_file.gz';
open my $fh, "gzip -cd $file |"
    or die "Couldn't read/decompress '$file': $!";
while (<$fh>) {
    ...
};
[download]

This approach also lends itself well to using the flip-flop operator .., see perlop.

open FILE, '>log_'.$expt_name.'.txt' or die "Can't open file : $!\n";
print FILE "***********************\n";
my @Ids=`zcat $log_files  | sed -e 's/\\\n/\\n/g' | grep -B5 'name\?'|
+ grep Id`;
foreach (@Ids){
print $_."\n";
#Search and print the first occurence of <score> after $_
}
close FILE;
[download]

There is not much code I have written so far :), trying to figure out which tools to use..thanks

While I'm not sure why you seem to write a shell script in Perl instead of using the Perl built-in mechanisms, I think if you post about 10 relevant lines of your input data. Your current approach should work regardless of whether the file is compressed or not, so I'm not sure I've understood where you see your problem.

I would unzip the file to a temporary location, then process it as a normal file.

This is the sort of thing that you could easily do with the awk tool, or of course with Perl.   The general approach that I would use is that of a state-machine.

In this approach, consider that there are exactly four kinds of lines you can be looking at, at any time:

1. Pattern 1.
2. Pattern 2.
3. A line that is neither.
4. End of file.   (No more lines exist.)

1. The last pattern seen was Pattern 1.
2. The last pattern seen was Pattern 2.
3. Neither pattern has been seen yet.  (Initial state.)

So, the general idea is to imagine a 3x4 rectangular table and to work out, for each square, what the program needs to do.

That is a very methodical approach and I will try the approach and post my code once its done. If I understand it correctly, this involves unzipping the file and using a file pointer to remember where I last stopped scanning. Is there any other way of doing it without unzipping the file? I am not averse to using awk/sed but from my limited knowledge, I cant seem to come up with a way of achieving this. The point I am getting struck is remmebering the position of pattern 1. Thats why I thought perl will rescue me with file handlers. But anything with file handlers will impose a limitation on the size of file I am dealing with. Also, I would like to point out that each of pettern 1 and pattern 2 are different.

< name > name 1 < /name >

#unknoen no of lines

< id > unique id1 < /id >

#unknow no of lines

< name > name 2 < /name >

#unknoen no of lines

< id > unique id2 < /id >

Hi Monks, I need suggestions on the fastest way to do this: I have 3 directories with log files in them. All log files have the following pattern.

< block>

< id> xyz < /id>

< url> foo.com < /url>

..

< response> xyz < /response>

< /block>

< block>

..

The task is to get the id if the url is foo.com from logs of first directory , search for that id in all the directories(including first one) print the responses from the corresponding blocks into a saparate file.

#getting the ids from first directory
sub doFile($) {
        my ($fn) =@_;
        chomp($fn);
        print "opening $fn\n";
        my $fh = IO::File->new($fn, 'r');
        my @msgLines;
      if( defined $fh){
        while(my $l = <$fh>) {
                push @msgLines, $l;
                if($l =~ m"</msg>\s*\$") {
                        #my $msg = join('', @msgLines);
                        my $id;
                        if(grep{ m"http://.*foo.com" } @msgLines) {
                              #store the @msglines into an array, this
+ array can serve as source for searching for reponses from first dire
+ctory, need to do something similar for the rest of directories.     
                                $id = grep { $_ =~ m"<Id>(\d+)</Id>"; 
+}
                         #@msgLines;
                          #      $id =~ m"<Id>(\d+)</Id>";
                                push @IDs, $id;
                                }
                        @msgLines = ();
                                          }
                           }
                       }
      else{
            die "Cannot open file $!\n";}
}
my @firstdir=@{$logfiles[0]};
my $path=$logdirs[0];
foreach (@firstdir) {
   my $curpath=sprintf($path.'/'.$_);
    print"In foreach trying to open $path\n";
        doFile($curpath);
}
[download]

The log files are huge, so zipping them into a single file is not possible(out of disk space). Any perl modules that can help me with this task?

#!/usr/bin/perl

use strict;
use PerlIO::gzip;

open( my $zip, "<:gzip", "bigfile.gz ) or die "$!\n";
while (<$zip>) {
    print if ( /some regex/ );
}
[download]

gunzip < bigfile.gz | grep 'some regex'
[download]

UPDATE: (2010-10-18) It seems that PerlIO::gzip should be viewed as superseded by PerlIO::via:gzip. (see PerlIO::gzip or PerlIO::via::gzip).

#!/usr/bin/perl

use strict;
use warnings;

my $lines = 0;
my $filename = '/path/to/ImageMagick.tar.gz';
die "Can't open '${filename}': $!" unless open(FILE, '<', $filename);
while (sysread FILE, my $buffer, 4096) {
               $lines += ($buffer =~ tr/\n//);
}
print $lines, "\n";
[download]

But a gzipped file is binary, and counting the newlines in a binary file does not make much sense.