Re^4: Threads From Hell #2: How To Parse A Very Huge File

For the grep example 60+60+42=162 which is 2m42s. But user+sys (20+5) is 0m25s. What do i miss?

My interpretation of those numbers is that difference of 137 seconds is the elapsed time when the processor is doing nothing (for this process) because the process is in the scheduler queue in an IO wait state, waiting for the disk.

That's when the opportunities for saving through multiprocessing simply don't exist. (As I detailed in my first reply above.)

It's also where marioroy's examples defy my analysis; because his system has the fastest IO rate I've ever seen. When my customer was planning to install PCIe SSDs in his server farm -- which seems like last week, but looking back was over 20 months ago -- the fastest commodity priced (he needed lots of them) cards available were Fusion-IO ioXtreme Pro 4-lane cards which were capable of something like 2.1Gbits/s (remember divide by at least 8 for GBytes/s). I guess that (somewhat) justifies the premium prices you pay for Apple hardware.

Interpreting those real/user/sys numbers gets further complicated when the elapsed time is less than the combined user+sys time, which comes about when multiple cores are processing concurrently, thus the process is racking up 4 (number of cores) seconds of cpu for every one second of elapsed time.

Then the waters get really muddy, when the IO waits on 4 cores, start balancing out the 4 seconds/second of cpu accumulations when there is processing to be done, and you end up with numbers that make it look like your sometimes-IO-bound/sometimes-CPU-bound process is doing 1 for 1, cpu to real seconds; but actually requires the use of 4 cores to achieve it.

Do you remember when I said a few days ago that it was very hard to draw generic conclusions about multi-threading ... :)

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". I'm with torvalds on this

In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

Comment on Re^4: Threads From Hell #2: How To Parse A Very Huge File

Replies are listed 'Best First'.
Re^5: Threads From Hell #2: How To Parse A Very Huge File by karlgoethebier (Abbot) on May 26, 2015 at 10:35 UTC
"...the elapsed time when the processor is doing nothing..." Yes. I asked the oracle: "The term "real time" in this context refers to elapsed "wall clock" time, like using a stop watch. The total CPU time (user time + sys time) may be more or less than that value. Because a program may spend some time waiting and not executing at all (whether in user mode or system mode) the real time may be greater than the total CPU time. Because a program may fork children whose CPU times (both user and sys) are added to the values reported by the time command, but on a multicore system these tasks are run in parallel, the total CPU time may be greater than the real time." "...justifies the premium prices you pay for Apple hardware." Did you already order your first Mac ;-) "...Do you remember when I said a few days ago..." Indeed. Best regards, Karl Edit: Fixed typo. P.S.: I'm thinking already about episode #3 of "Threads From Hell". «The Crux of the Biscuit is the Apostrophe»	[reply]
Re^6: Threads From Hell #2: How To Parse A Very Huge File by BrowserUk (Patriarch) on May 26, 2015 at 10:53 UTC
Did you already order your first Mac I did say "somewhat". And no. There is nothing in this world that would persuade me to buy an unrepairable, unexpandable, fashion accessory computer with a 100%+ idiot tax. P.S.: I'm thinking already about episode #4 of "Threads From Hell"./i> Did I miss the 3rd installment? With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked	[reply]
Re^7: Threads From Hell #2: How To Parse A Very Huge File by karlgoethebier (Abbot) on May 26, 2015 at 19:47 UTC
"...an unrepairable, unexpandable, fashion accessory computer..." Yes - less or more. But take a look at this snows of yesteryear: "...the world’s most powerful and cheapest homebuilt supercomputer." See also System X. Regards, Karl «The Crux of the Biscuit is the Apostrophe»	[reply]
Re^8: Threads From Hell #2: How To Parse A Very Huge File by marioroy (Prior) on May 29, 2015 at 12:13 UTC
Re^9: Threads From Hell #2: How To Parse A Very Huge File by BrowserUk (Patriarch) on May 29, 2015 at 13:34 UTC


P is for Practical
	PerlMonks