good chemistry is complicated, and a little bit messy -LW |
|
PerlMonks |
How to read in large filesby Only1KW (Sexton) |
on Jan 28, 2016 at 15:34 UTC ( [id://1153883]=perlquestion: print w/replies, xml ) | Need Help?? |
Only1KW has asked for the wisdom of the Perl Monks concerning the following question: I am attempting to parse a large file (~10 GB) but having issues with running out of system memory to do so. I start out with about 3.5 GB used and end up using all 16 GB before Linux kills the process. Here is the code that is failing for me:
When I run this program, "Start Read" is printed to the screen, but I never see "Read Line". I've done a bunch of Googling on this, and found a bunch of hits, but everything I read says to just read the file one line at a time to get around the size. But isn't that what I'm already doing??? All the specific examples I see for fixing the issue focus on other things the user is also doing which are also taking up memory.
Back to
Seekers of Perl Wisdom
|
|