Re^2: Out of Memory Error : V-Lookup on Large Sized TEXT File

This code will not compile - not enough code shown.

use strict;
use warnings;
# REFFILELIST is not the same as $ReferenceFilePath
[download]

What op said:

open (FILE, $ReferenceFilePath) or die "Can't open file";
chomp (@REFFILELIST = (<FILE>));
[download]

The correct way is to iterate over the opened input largefile file handle.

while (<FILE>)
{
  chomp;
  #do something
}
[download]

Comment on Re^2: Out of Memory Error : V-Lookup on Large Sized TEXT File Select or Download Code

Replies are listed 'Best First'.
Re^3: Out of Memory Error : V-Lookup on Large Sized TEXT File by Laurent_R (Canon) on Apr 28, 2015 at 06:17 UTC
`$ReferenceFilePath` is the name of the file, `FILE` the file handler opened on this file and `@REFFILELIST` is the array in which to store the lines of this file. Even though I would rather use lexical file handler and three-argument syntax for open, I do not see the syntax of this part of the code to be really wrong (just outdated and slightly deprecated). I also do not consider storing reference data into an array to be wrong (a hash might be better, but we don't know enough about the data to be sure). But storing the data in an array and then looping on that array is not good, it would be better to iterate on the lines. My view is that it makes sense to store the reference data in memory if you then iterate on the large file and use the in-memory data structure to look up for something. But again, we don't know enough about the data and about the real intent of the program. Je suis Charlie.	[reply] [d/l] [select]
Re^4: Out of Memory Error : V-Lookup on Large Sized TEXT File by Marshall (Canon) on Apr 30, 2015 at 03:59 UTC
I think that you are basically agreeing with me, quibbles about syntax aside. I was reacting to the idea of "out of memory". That will not happen if the huge file is processed line by line after the FILE open instead of being stored as an array and then processed in memory. There could be some other problem here that the OP did not show and is completely unknowable to us because it wasn't shown. A 100MB file even under Win32, version XP should be easy to do, even as an in memory array. A FILE handle should not be "re-opened" with an append for multiple writes - that can cause memory problems with the file system. Keep the file handle open and just continue writing.	[reply]