Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re: performance of File Parsing

by BrowserUk (Patriarch)
on Jul 07, 2011 at 08:45 UTC ( [id://913152]=note: print w/replies, xml ) Need Help??


in reply to performance of File Parsing

I have the tera byte file ...

Method 1: Will work.

Method 2: Won't work. (Or will be horribly slow.)

Although a few of the latest 64-bit processors can theoretically address 1TB of memory, most motherboards are limited to much less. Even the top end, SMP & NUMA processors boards and cards max out at figures of 64/128/256GB of physical memory.

Whilst it is possible to use swap files to extend the virtual memory available to a process into the TB range, the effect on performance is dire. Instead of reading once, processing and writing result, you (minimally) end up: read from file, write to swap, read from swap, process, write result to disk. Ie. You must do 4 IO ops instead of two and that will at least double your processing time, and usually much worse.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://913152]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others chilling in the Monastery: (4)
As of 2024-04-19 15:48 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found