Hmm, your benchmark is very interesting in terms of comparing various ways of storing an array into a hash, and I'll definitely keep the data somewhere for my own benefit, but I am not sure this benchmark is really relevant to the OP, which was about reading a file into memory. If the I/O take much more time than working in memory, then, where you have, say, a 20% performance improvement with an array into a hash, it might only be a 2% performance improvement when reading a file is involved, and the improvement is probably really not worth the trouble in this case.
My point in my previous post was about avoiding using too much memory, rather than CPU usage. I am working daily with very large files, and quite often with really huge ones. Most of the time, I do not care too much whether my program processing 100 million records will run in 10 or in 20 minutes, but I do really care whether it will go to completion or blow up for lack of memory. My work is very often to compare two very large files. Quite often, the data volume will simply not fit in memory. My strategy in such cases is often to first sort the files according to some unique key (for example with the Unix sort utility), and then to compare them line by line (which is not as easy as it might look, reading two files in parallel is not so easy when you may have missing lines on one of the file or the other). But once you've got the algorithm right, this is really very fast. Well, I might have been carried away, I just wanted to say that CPU usage is not necessarily the ultimate goal, sometimes memory usage is far more important (at least when it make the difference between a program that dies before completion or that goes smoothly to the end).
In reply to Re^3: get difference of two arrays
by Laurent_R
in thread get difference of two arrays
by naturalsciences
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |