Re^3: Out of Memory Error : V-Lookup on Large Sized TEXT File

$ReferenceFilePath is the name of the file, FILE the file handler opened on this file and @REFFILELIST is the array in which to store the lines of this file. Even though I would rather use lexical file handler and three-argument syntax for open, I do not see the syntax of this part of the code to be really wrong (just outdated and slightly deprecated).

I also do not consider storing reference data into an array to be wrong (a hash might be better, but we don't know enough about the data to be sure). But storing the data in an array and then looping on that array is not good, it would be better to iterate on the lines. My view is that it makes sense to store the reference data in memory if you then iterate on the large file and use the in-memory data structure to look up for something. But again, we don't know enough about the data and about the real intent of the program.

Je suis Charlie.

Comment on Re^3: Out of Memory Error : V-Lookup on Large Sized TEXT File Select or Download Code

Replies are listed 'Best First'.
Re^4: Out of Memory Error : V-Lookup on Large Sized TEXT File by Marshall (Canon) on Apr 30, 2015 at 03:59 UTC
I think that you are basically agreeing with me, quibbles about syntax aside. I was reacting to the idea of "out of memory". That will not happen if the huge file is processed line by line after the FILE open instead of being stored as an array and then processed in memory. There could be some other problem here that the OP did not show and is completely unknowable to us because it wasn't shown. A 100MB file even under Win32, version XP should be easy to do, even as an in memory array. A FILE handle should not be "re-opened" with an append for multiple writes - that can cause memory problems with the file system. Keep the file handle open and just continue writing.	[reply]

Replies are listed 'Best First'.

Re^4: Out of Memory Error : V-Lookup on Large Sized TEXT File
by Marshall (Canon) on Apr 30, 2015 at 03:59 UTC

I was reacting to the idea of "out of memory". That will not happen if the huge file is processed line by line after the FILE open instead of being stored as an array and then processed in memory.

There could be some other problem here that the OP did not show and is completely unknowable to us because it wasn't shown. A 100MB file even under Win32, version XP should be easy to do, even as an in memory array. A FILE handle should not be "re-opened" with an append for multiple writes - that can cause memory problems with the file system. Keep the file handle open and just continue writing.

[reply]