Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I've seen many post about reading large files efficiently but has anyone experienced a error when trying to open a large file say (3 - 4 gigs)? The code below works fine on small files but when i try to parse a 3 - 4 gig file i get this error: readline() on closed filehandle main::IN at ./parse.pl line 11. 
#!/usr/bin/perl -w 

$| = 1;

$in_file = 'very_large_text_file'; 
$out_file = 'parsed_text_file';

open(IN, "<$in_file");
open(OUT, ">$out_file");

while(<IN>){
  ($email, $other) = split(/\s+/,$_); 
  print OUT "$email\n";
}

close(OUT);
close(IN);

[download]

Replies are listed 'Best First'.
Re: Parsing a 3 - 4 gig text file
by VSarkiss (Monsignor) on May 14, 2002 at 19:40 UTC

    It would help to know what the error is on the open. Check for that, and print a message if it fails, like so: 

    open(IN, "<$in_file")
    or die "Can't open $in_file: $!\n";

    [download]
    Clearly the file isn't open by the time you try to read it, so start by figuring that one out first.

[reply]
[d/l]
[select]
      The error is: File too large at: $lineNumber
[reply]
        That's a sign that you have a perl that's not built with large file support. You'll need to rebuild your perl to access that file, or play games with pipes and cat and suchlike things.
[reply]
Re: Parsing a 3 - 4 gig text file
by grep (Monsignor) on May 14, 2002 at 19:44 UTC
[reply]
      I know this sounds crazy, but have you checked the syntax of the name of the large file you are trying to read? In my past experiences with large fle manipulation, I have only recieved that error when I goof, and actually try to read on a closed file(don't ask, heh) or if the file I'm trying to read from simply doesn't exist(i.e. Typo on the file name). I think that if the data your trying to read from this file was too big, it would give a different error.

      Am I making sence?

      -- Yes, I am a criminal. My crime is that of defyance.
[reply]

Back to Seekers of Perl Wisdom