comment on

I have the tera byte file ...

Method 1: Will work.

Method 2: Won't work. (Or will be horribly slow.)

Although a few of the latest 64-bit processors can theoretically address 1TB of memory, most motherboards are limited to much less. Even the top end, SMP & NUMA processors boards and cards max out at figures of 64/128/256GB of physical memory.

Whilst it is possible to use swap files to extend the virtual memory available to a process into the TB range, the effect on performance is dire. Instead of reading once, processing and writing result, you (minimally) end up: read from file, write to swap, read from swap, process, write result to disk. Ie. You must do 4 IO ops instead of two and that will at least double your processing time, and usually much worse.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

In reply to Re: performance of File Parsing by BrowserUk
in thread performance of File Parsing by Anonymous Monk

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


Perl: the Markov chain saw
	PerlMonks