How can I process large files with while(<>)?

Ritter has asked for the wisdom of the Perl Monks concerning the following question:

When using the code below for modify each line of a large textfile (100MB) I get the message "Out of memory" after some time. I am told this is because @file = <HANDLE>; opens the entire file at once instead of reading one line at the time with while(<>).

How do I use while(<>) to manage the same task as below but for large files by reading line by line?

open (HANDLE, "list.txt") or die "can't open file";
@file = <HANDLE>;
foreach $row (@file) { 
open NEWFILELIST, ">>newlist.txt" or die "Cannot open newlist.txt for 
+append :$!";
print NEWFILELIST "outputting modified line from list.txt\n";
close NEWFILELIST;
 }
close (HANDLE);
[download]

Thank you,
Ritter

Comment on How can I process large files with while(<>)? Select or Download Code

Replies are listed 'Best First'.
Re: How can I process large files with while(<>)? by rob_au (Abbot) on Nov 17, 2002 at 09:18 UTC
How about this ... `open( HANDLE, "list.txt" ) or die "Cannot open file - $!"; while( <HANDLE> ) { # Process file line by line here } close HANDLE;` [download] The `while(<>)` structure iterates through the magic `ARGV` handle, line by line, where each line is defined as ending by the input record separator, `$/`. In the example above, the filehandle, `HANDLE` is specified to be iterated through in this manner thereby performing the task of reading through a large file without storing it all in an array first. `perl -e 'print+unpack("N",pack("B32","00000000000000000000000111100011")),"\n"'`	[reply] [d/l] [select]
Re: Re: How can I process large files with while(<>)? by Ritter (Sexton) on Nov 17, 2002 at 10:09 UTC
Working perfectly! thanks! Ritter	[reply]
Re: How can I process large files with while(<>)? by mattr (Curate) on Nov 17, 2002 at 10:11 UTC
Yup. Also you can do this on the command line using things like perl -nle (make an input loop) or perl -pi -e ("in-place" editting) as described in the perlrun manpage. Since my paranoia turns on when I use -pi I often turn off paranoia (a biological process) and do something like `cat bigfile\|perl -nle 's/escaped_old_ip/new_ip/g; print;' > newbigfile` You can also process files using the tie command, there are many modules which use it (Tie::). Good luck!	[reply] [d/l]
Re^2: How can I process large files with while(<>)? by Aristotle (Chancellor) on Nov 18, 2002 at 00:01 UTC
For one, that is a useless use of `cat`, for another, why use `-n` when you just want to print your lines anyway? That's what `-p` is for. You can also put `-i` to good use here to edit the file inplace, as well as supply a suffix for a backup copy to be kept. `perl -i.bak -pe 's/escaped_old_ip/new_ip/g' bigfile` Details are available in said manpage. `:^)` Makeshifts last the longest.	[reply] [d/l]
Re: How can I process large files with while(<>)? by pg (Canon) on Nov 17, 2002 at 16:43 UTC
You got the answer already, but I just want to mention one point. This is actually one of the most sweet thing in Perl, a Perl function can work under different context, it understand what you want, and gives what you want, as long as you understand it well. When the return might be in different forms, you can see it just fits in the context/situation so well. So, when you learn a func, always try to understand this aspect of it.	[reply]