in reply to Help needed in reading a very large file line by line

The example you linked to deals with files with fixed-length records - i.e. generally binary files. As you're talking about "line by line", I assume you're talking about a text-based file - e.g. plain text, HTML, CSV, etc.

# Set the character which will be used to indicate the end of a line. # This defaults to the system's end of line character, but it doesn't # hurt to set it explicitly, just in case some other part of your code # has altered it from the default. local $/ = "\n"; # Open the file for read access: open my $filehandle, '<', 'myfile.txt'; my $line_number = 0; # Loop through each line: while (defined($line = <$filehandle>)) { # The text of the line, including the linebreak # is now in the variable $line. # Keep track of line numbers $line_number++; # Strip the linebreak character at the end. chomp $line; # Do something with the line. do_something($line); # Perhaps bail out of the loop if ($line =~ m/^ERROR/) { warn "Error on line $line_number - skipping rest of file"; last; } }

But we can make the above more concise, because Perl usefully defines a variable called $_ which is used as a default variable in many cases; and a variable called $. which keeps track of the current line number.

# Set the character which will be used to indicate the end of a line. local $/ = "\n"; # Open the file for read access: open my $filehandle, '<', 'myfile.txt'; # Loop through each line: while (<$filehandle>) { # The text of the line, including the linebreak # is now in the variable $_. # Strip the linebreak character at the end. chomp; # Do something with the line. do_something($_); # Perhaps bail out of the loop if (m/^ERROR/) { warn "Error on line $. - skipping rest of file"; last; } }

Replies are listed 'Best First'.
Re^2: Help needed in reading a very large file line by line
by AnomalousMonk (Archbishop) on Feb 28, 2012 at 22:07 UTC
    The example you linked to deals with files with fixed-length records...

    NB: Re: help reading from large file needed begins by briefly alluding to processing files with fixed-length records, but then continues with a detailed discussion, with example, of indexing a variable-length record file for rapid random access.

Re^2: Help needed in reading a very large file line by line
by Anonymous Monk on Feb 28, 2012 at 10:59 UTC
    while(<FILEHANDLE>) is giving an out of memory error.

      Just post your script (between <c> </c> tags) and we can tell you what is wrong.

      Chances are that either $/ is set to something silly instead of "\n", or your file has no line break characters in it (or at least, very long lines).

      Hi, I see this is an old thread, but still, I would like to share that I have expereciend something similar. I wanted to do a very simple search and replace on a huge ASCII file (around 4GB) using the magic filehandle <>. The thing is that I cannot use seek or whichever method that requires fixed length of records. Also my $/ is set to "\n" and I know that the lines are not incredibly long. Any ideas?

      Here is a piece of code:

      my $fh = new FileHandle; @ARGV = ($file); open $fh, ">test.txt"; while ($line = <>) { $line =~ s/$search/$replace/g; print $fh $line; }
        Can you
        print "$.: $line"
        and check at what line it fails?
        لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ