No such thing as a small change | |
PerlMonks |
Re: performance of File Parsingby BrowserUk (Patriarch) |
on Jul 07, 2011 at 08:45 UTC ( [id://913152]=note: print w/replies, xml ) | Need Help?? |
I have the tera byte file ... Method 1: Will work. Method 2: Won't work. (Or will be horribly slow.) Although a few of the latest 64-bit processors can theoretically address 1TB of memory, most motherboards are limited to much less. Even the top end, SMP & NUMA processors boards and cards max out at figures of 64/128/256GB of physical memory. Whilst it is possible to use swap files to extend the virtual memory available to a process into the TB range, the effect on performance is dire. Instead of reading once, processing and writing result, you (minimally) end up: read from file, write to swap, read from swap, process, write result to disk. Ie. You must do 4 IO ops instead of two and that will at least double your processing time, and usually much worse. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
In Section
Seekers of Perl Wisdom
|
|