in reply to performance of File Parsing
I have the tera byte file ...
Method 1: Will work.
Method 2: Won't work. (Or will be horribly slow.)
Although a few of the latest 64-bit processors can theoretically address 1TB of memory, most motherboards are limited to much less. Even the top end, SMP & NUMA processors boards and cards max out at figures of 64/128/256GB of physical memory.
Whilst it is possible to use swap files to extend the virtual memory available to a process into the TB range, the effect on performance is dire. Instead of reading once, processing and writing result, you (minimally) end up: read from file, write to swap, read from swap, process, write result to disk. Ie. You must do 4 IO ops instead of two and that will at least double your processing time, and usually much worse.
|
|---|