Re: Out of memory error while Reading 5GB text file

Take a look at the Tie::File module:

Tie::File represents a regular text file as a Perl array. Each element in the array corresponds to a record in the file. The first line of the file is element 0 of the array; the second line is element 1, and so on.
The file is not loaded into memory, so this will work even for gigantic files.

Jason L. Froebe

Help find a cure for breast cancer! Net proceeds benefit the Susan G. Komen Breast Cancer Foundation and the National Philanthropic Trust. Help by donating - I'm walking 60 miles in 3 days in August 2007. (The day I return from TechWave is the first day of the Walk).

Blog, Tech Blog

Comment on Re: Out of memory error while Reading 5GB text file

Replies are listed 'Best First'.
Re^2: Out of memory error while Reading 5GB text file by ikegami (Patriarch) on Apr 04, 2007 at 15:57 UTC
Tie::File keeps an index of every line it encounters (which is every line up to the deepest one requested). For a 5GB file, that's a giant amount of memory. The `memory` option does not affect the size of this index.	[reply] [d/l]