Re: Reading from file, not to memory

When you reference a file handle in list context (by assigning to an array), you read the whole file at once into memory. You can intead read the file line by line by using the file handel in scalar context (by assigning to a scalar). Instead of this...

open(FH, "/path/to/file") || die "Can't open file: $!";
my @file = <FH>;
close FH;
[download]

...try this...

open(FH, "/path/to/file") || die "Can't open file: $!";
foreach my $file (<FH>){
     if($file eq $external_info){
          print "great\!";
          last;
     }
}
close FH;
[download]

---
print map { my ($m)=1<<hex($_)&11?' ':'';
$m.=substr('AHJPacehklnorstu',hex($_),1) }
split //,'2fde0abe76c36c914586c';
[download]

Comment on Re: Reading from file, not to memory Select or Download Code

Replies are listed 'Best First'.
Re: Re: Reading from file, not to memory by cees (Curate) on Feb 20, 2003 at 04:33 UTC
You have to be careful here, because you are doing exactly the same thing. The `foreach` statement is also accessing <FB> in array context and is reading the whole file into an anonymous array and then iterating over it. You have to use a `while` loop for this to work correctly. Try out the following bit of code to test this: `use Benchmark; timethese(1, { 'Trial1 While' => sub { open (FILE, "file2") or die "Can't open file: $!\n"; while (<FILE>) { last; # read one line and exit } close FILE; }, 'Trial2 Foreach' => sub { open (FILE, "file1") or die "Can't open file: $!\n"; foreach my $line (<FILE>) { last; # read one line and exit } close FILE; }, });` [download] Make sure that file1 and file2 are identical (I used two files so that we know there is no caching going on), and that they are large text files. I got the following results with 2 50Meg files: `Benchmark: timing 1 iterations of Trial1 While, Trial2 Foreach... Trial1 While: 0 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU) Trial2 Foreach: 17 wallclock secs (10.77 usr + 1.56 sys = 12.33 CPU) +@ 0.08/s (n=1)` [download]	[reply] [d/l] [select]
Re: Re: Reading from file, not to memory by gmpassos (Priest) on Feb 20, 2003 at 08:02 UTC
Your code still loads all the file first! The code (<FH>) generate first all the array that will be readed by foreach! Take a look in this 2 test scripts. They show the file buffer position when you print each line: `my $file = $0 ; open(FH, $file) ; foreach my $line (<FH>){ my $tel = tell(FH) ; print "$tel>> $line" ; } close FH;` [download] OK way: `my $file = $0 ; open(FH, $file) ; while ( my $line = <FH> ) { my $tel = tell(FH) ; print "$tel>> $line" ; } close FH;` [download] You can see that in the 1st way the buffer was always in the end, because it already have loaded all the file! Graciliano M. P. "The creativity is the expression of the liberty".	[reply] [d/l] [select]
Re: Re: Reading from file, not to memory by pfaut (Priest) on Feb 20, 2003 at 13:51 UTC
Let this note stand as a testimony to the dangers of cut-n-paste or cargo culting. In my code, I always use `while` when reading from files (honest!) but in my laziness, I copied and pasted from the base node into my reply without looking the code over well enough. `--- print map { my ($m)=1<<hex($_)&11?' ':''; $m.=substr('AHJPacehklnorstu',hex($_),1) } split //,'2fde0abe76c36c914586c';` [download]	[reply] [d/l] [select]