Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,
I am trying to read 5GB(file size) text file and search some content in that file. But I got "Out of memory" error.
Could you please provide your suggestion. 

open FIN, $inputFile or die $!;
while (my $lines  = <FIN>) {
   chomp $sword;
   unless ($lines =~ /$sword/i) {
        print "$sword\n";
   }
}
close FIN;

[download]

thanks,
PerlUser.

Replies are listed 'Best First'.
Re: Out of memory error while Reading 5GB text file
by varian (Chaplain) on Apr 04, 2007 at 06:59 UTC
    Normally to read a textfile in this fashion should be just fine. However my hinch is that the code example that you presented is incomplete: 
    while (my $lines  = <FIN>) {
   chomp $sword;

    [download]
    e.g. I would have expected a chomp $lines within the loop. Maybe show us some more code details?

    Now if you accidently would parse a file that is not a textfile the inputline may end up being too long and on some systems you could run out of memory.

    One cause for the above could be platform differences i.e. if you read on an Windows platform a file originally created on a Unix platform the textline separator might be different and your program may fail to recognise end-of-line.
    Note that this problem would not be as bad if the source is Windows and the destination is Unix. In that case you just find an extra "\r" as the last character on the line after you have chomped the input.

[reply]
[d/l]
Re: Out of memory error while Reading 5GB text file
by idle (Friar) on Apr 04, 2007 at 06:45 UTC
[reply]
Re: Out of memory error while Reading 5GB text file
by prasadbabu (Prior) on Apr 04, 2007 at 06:34 UTC
[reply]
Re: Out of memory error while Reading 5GB text file
by robot_tourist (Hermit) on Apr 04, 2007 at 14:04 UTC

    where is $sword set? From that fragment, it looks like you're not chomp()ing anything.

    How can you feel when you're made of steel? I am made of steel. I am the Robot Tourist.
    Robot Tourist, by Ten Benson

[reply]
Re: Out of memory error while Reading 5GB text file
by jfroebe (Parson) on Apr 04, 2007 at 13:41 UTC

    Take a look at the Tie::File module:

    Tie::File represents a regular text file as a Perl array. Each element in the array corresponds to a record in the file. The first line of the file is element 0 of the array; the second line is element 1, and so on.
    The file is not loaded into memory, so this will work even for gigantic files.

    Jason L. Froebe

    Help find a cure for breast cancer! Net proceeds benefit the Susan G. Komen Breast Cancer Foundation and the National Philanthropic Trust. Help by donating - I'm walking 60 miles in 3 days in August 2007. (The day I return from TechWave is the first day of the Walk).

    Blog, Tech Blog

[reply]
      Tie::File keeps an index of every line it encounters (which is every line up to the deepest one requested). For a 5GB file, that's a giant amount of memory. The memory option does not affect the size of this index.
[reply]
[d/l]
A reply falls below the community's threshold of quality. You may see it by logging in.

Back to Seekers of Perl Wisdom