out of memory error while reading file contents into an array

psr473 has asked for the wisdom of the Perl Monks concerning the following question:

I have input raw files size in GB. I need to create small files with the content based on some search criteria. For this I have opened file and reading the contents of input file into array. At the time reading I am getting Out Memory Error (on both Windows & UNIX). Currently I am trying on Windows. Below is the code:

open (IN, filename) or die "cann't open file filename : $!\n";
my @arr = <IN>;
close (IN);
[download]

please help me to get rid of this error. Thank you in advance for your help.

Comment on out of memory error while reading file contents into an array Download Code

Replies are listed 'Best First'.
Re: out of memory error while reading file contents into an array by tokpela (Chaplain) on Aug 11, 2009 at 14:03 UTC
You should iterate over the filehandle directly. `use strict; use warnings; my $filename = 'filename.txt'; open(my $fh, $filename) or die "cann't open file filename : $!\n"; while (<$fh>) { my $line = $_; next if ($line =~ /^\s*$/); # skip blank lines chomp($line); # remove line endings # .... do your coding here on each line } close($fh);` [download]	[reply] [d/l]
Re: out of memory error while reading file contents into an array by kennethk (Abbot) on Aug 11, 2009 at 13:52 UTC
Please read Markup in the Monastery; in particular, please do not use <pre> tags in your write-ups, as they screw up many people's display and generally make grumpy monks. If you don't have space to hold the entire file in memory, then don't read in the entire file - process line by line. `open (IN, filename) or die "cann't open file filename : $!\n"; while (defined(my $line = <IN>)) { #process $line } close (IN);` [download]	[reply] [d/l]
Re^2: out of memory error while reading file contents into an array by psr473 (Novice) on Aug 12, 2009 at 11:41 UTC
Thanks for your help. I will try line by line.	[reply]
Re: out of memory error while reading file contents into an array by ssandv (Hermit) on Aug 11, 2009 at 15:32 UTC
To provide a little more detail on what's happening here, your code, as written, reads the entire file in at once. If the file is in GB, you'll need GB of memory dedicated to that process just to handle the data--not counting the memory consumed by the OS, Perl itself, and so forth. As others have suggested, you need to iterate over the filehandle, so you read one line at a time, and operate on it. You may need an intermediate step of writing the small files out unsorted, then rereading them to sort them, if they need internal sorting, since you can't hold the entire dataset in memory. Additionally, you would be better off to use 3-argument open and a scalar filehandle: `open my $input_fh, '<', $filename or die $!;` [download] unless you're absolutely certain your filename string is free of characters that could be misinterpreted by 2-argument open (and frankly, it's a good habit to be in anyway.) The scalar filehandle gives you scoping, which bareword filehandles don't really have. (And, as others have also pointed out, you should follow the markup directions right below the box you write your post in that say not to use `<pre>` tags.)	[reply] [d/l] [select]
Re: out of memory error while reading file contents into an array by Bloodnok (Vicar) on Aug 11, 2009 at 13:49 UTC
Hmmm, obviously there was enough memory to include _all_ your posting within <code> tags - did the preview not look at all odd to you - try putting just the actual code in the tags!! A user level that continues to overstate my experience :-))	[reply]
Re: out of memory error while reading file contents into an array by JavaFan (Canon) on Aug 11, 2009 at 13:43 UTC
Buy more memory.	[reply]