soubalaji has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

Iam reading a GB size file in unix box. While reading the file iam getting a Out of Memory Issue. Below is the code:

open(INFILE, "zgrep -dc $myfile");

while (<INFILE>){

if (s/match//){

replacing;

}

}

close INFILE;

Can any one suggest me how to handle this.

Replies are listed 'Best First'.
Re: Out of Memory
by NetWallah (Canon) on Jun 04, 2011 at 05:20 UTC
    Your "open" is probably failing silently, and it is doubtful if your "Out of memory" is in this code.

    Perhaps , what you intended was:

    open(INFILE, "-|", "zgrep -dc $myfile") or die "Could not zgrep $myfil +e : $!";

                "XML is like violence: if it doesn't solve your problem, use more."

Re: Out of Memory
by Khen1950fx (Canon) on Jun 04, 2011 at 11:25 UTC
    I think that what's happening to you is that your command is missing -e EXPR. Without EXPR, the command will start to run and forget to finish, slowing things down until the whole computer grinds to a halt.

    The trick to zgrep is that it really isn't very portable. For example, the -d option isn't supported by my system's grep. Here's what I used that worked:

    #!/usr/bin/perl use strict; use warnings; my $file = shift @ARGV; open(INFILE, "|-", "zgrep -c -e / $file ") or die $!; close INFILE;
    You'll notice that I used "|-": it writes to STDOUT, which is what I think that you were trying to do. I also tried it with "-|": it writes to STDIN, and that didn't work.
      Your example sends zgrep's output directly to STDOUT, avoiding the need to read it using "while (<INFILE>)".

      I think the OP's intent is to read and process the output of zrep, so, he should use "-|", and read it using "while(<INFILE>)".

                  "XML is like violence: if it doesn't solve your problem, use more."

Re: Out of Memory
by deep3101 (Acolyte) on Jun 04, 2011 at 00:52 UTC
    What is the size of RAM you have on your PC. Also i would like to ask did you try importing the contents of the file to a variable scalar or array like  @a=<INFILE>; or like $b=<INFILE> .
      Iam running this script in the Unix server. And i tired to store that in the variable ($b=<INFILE>). But no result.