For the grep example 60+60+42=162 which is 2m42s. But user+sys (20+5) is 0m25s. What do i miss?

My interpretation of those numbers is that difference of 137 seconds is the elapsed time when the processor is doing nothing (for this process) because the process is in the scheduler queue in an IO wait state, waiting for the disk.

That's when the opportunities for saving through multiprocessing simply don't exist. (As I detailed in my first reply above.)

It's also where marioroy's examples defy my analysis; because his system has the fastest IO rate I've ever seen. When my customer was planning to install PCIe SSDs in his server farm -- which seems like last week, but looking back was over 20 months ago -- the fastest commodity priced (he needed lots of them) cards available were Fusion-IO ioXtreme Pro 4-lane cards which were capable of something like 2.1Gbits/s (remember divide by at least 8 for GBytes/s). I guess that (somewhat) justifies the premium prices you pay for Apple hardware.

Interpreting those real/user/sys numbers gets further complicated when the elapsed time is less than the combined user+sys time, which comes about when multiple cores are processing concurrently, thus the process is racking up 4 (number of cores) seconds of cpu for every one second of elapsed time.

Then the waters get really muddy, when the IO waits on 4 cores, start balancing out the 4 seconds/second of cpu accumulations when there is processing to be done, and you end up with numbers that make it look like your sometimes-IO-bound/sometimes-CPU-bound process is doing 1 for 1, cpu to real seconds; but actually requires the use of 4 cores to achieve it.

Do you remember when I said a few days ago that it was very hard to draw generic conclusions about multi-threading ... :)


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

In reply to Re^4: Threads From Hell #2: How To Parse A Very Huge File by BrowserUk
in thread Threads From Hell #2: How To Search A Very Huge File [SOLVED] by karlgoethebier

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.