Re^2: Threads From Hell #2: How To Parse A Very Huge File

Okay. From your post:

$ time wc -l very_huge.file
100000001 very_huge.file

real  0m0.781s
user  0m0.521s
sys   0m0.260s
[download]

So, user code time: 0.521 + system code time: 0.260 = real time: 0.781. Makes sense!

But then:

$ time ./MCE-1.608/examples/wc.pl -l very_huge.file
100000001 very_huge.file

real  0m0.612s
user  0m0.633s
sys   0m1.522s
[download]

User code time: 0.633 + system code time: 1.522 = real time: 2.155s != 0.612s doesn't!

The other examples seem to indicate -- although the "timing information" is equally futzed -- that if you force the real wc to run more slowly, then your fake wc runs less, more slowly?

Is the lesson of your post that if you force wc to count characters -- which you can determine instantly from ls/dir -- and words -- which nobody is ever interested in; whilst also forcing it to use an 'unnatural' collating sequence that is equally of no interest, then your MCE::Grep can do what nobody wants done, less slowly than the system (which?) provided binary does, what nobody wants done?

Great advert.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". I'm with torvalds on this

In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

Comment on Re^2: Threads From Hell #2: How To Parse A Very Huge File Select or Download Code

Replies are listed 'Best First'.
Re^3: Threads From Hell #2: How To Parse A Very Huge File by karlgoethebier (Abbot) on May 25, 2015 at 00:07 UTC
"...User code time... + system code time... = real time..." Yes: From the docs: "The time utility executes and times the specified utility. After the utility finishes, time writes to the standard error stream, (in seconds): the total time elapsed, the time used to execute the utility process and the time consumed by system overhead." Some observations: `karls-mac-mini:monks karl$ ls -hl very_huge10GB.file -rw-r--r-- 1 karl karl 10G 25 Mai 00:53 very_huge10GB.file karls-mac-mini:monks karl$ time grep karl very_huge10GB.file nose cuke karl nose cuke karl nose cuke karl nose cuke karl nose cuke karl real 2m42.126s user 0m20.437s sys 0m5.645s` [download] `karls-mac-mini:monks karl$ ./mce_loop.pl nose cuke karl nose cuke karl nose cuke karl nose cuke karl nose cuke karl Took 150.555 seconds` [download] `#!/usr/bin/env perl use Time::HiRes qw( time ); use feature qw(say); my $start = time; say qx (grep karl very_huge10GB.file); printf "Took %.3f seconds\n", time - $start; __END__ karls-mac-mini:monks karl$ ./wrap.pl nose cuke karl nose cuke karl nose cuke karl nose cuke karl nose cuke karl Took 157.265 seconds` [download] For the grep example 60+60+42=162 which is 2m42s. But user+sys (20+5) is 0m25s. What do i miss? Perhaps it's too late tonight. Or too early in the morning? Best regards, Karl ŤThe Crux of the Biscuit is the Apostropheť	[reply] [d/l] [select]
Re^4: Threads From Hell #2: How To Parse A Very Huge File by marioroy (Prior) on May 25, 2015 at 00:28 UTC
For the grep example 60+60+42=162 which is 2m42s. But user+sys (20+5) is 0m25s. What do i miss? The wait time is not computed in the time output. This demonstration is IO bound. Basically, CPUs idle when waiting for IO.	[reply]
Re^4: Threads From Hell #2: How To Parse A Very Huge File by BrowserUk (Patriarch) on May 25, 2015 at 01:00 UTC
For the grep example 60+60+42=162 which is 2m42s. But user+sys (20+5) is 0m25s. What do i miss? My interpretation of those numbers is that difference of 137 seconds is the elapsed time when the processor is doing nothing (for this process) because the process is in the scheduler queue in an IO wait state, waiting for the disk. That's when the opportunities for saving through multiprocessing simply don't exist. (As I detailed in my first reply above.) It's also where marioroy's examples defy my analysis; because his system has the fastest IO rate I've ever seen. When my customer was planning to install PCIe SSDs in his server farm -- which seems like last week, but looking back was over 20 months ago -- the fastest commodity priced (he needed lots of them) cards available were Fusion-IO ioXtreme Pro 4-lane cards which were capable of something like 2.1Gbits/s (remember divide by at least 8 for GBytes/s). I guess that (somewhat) justifies the premium prices you pay for Apple hardware. Interpreting those real/user/sys numbers gets further complicated when the elapsed time is less than the combined user+sys time, which comes about when multiple cores are processing concurrently, thus the process is racking up 4 (number of cores) seconds of cpu for every one second of elapsed time. Then the waters get really muddy, when the IO waits on 4 cores, start balancing out the 4 seconds/second of cpu accumulations when there is processing to be done, and you end up with numbers that make it look like your sometimes-IO-bound/sometimes-CPU-bound process is doing 1 for 1, cpu to real seconds; but actually requires the use of 4 cores to achieve it. Do you remember when I said a few days ago that it was very hard to draw generic conclusions about multi-threading ... :) With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked	[reply]
Re^5: Threads From Hell #2: How To Parse A Very Huge File by karlgoethebier (Abbot) on May 26, 2015 at 10:35 UTC
"...the elapsed time when the processor is doing nothing..." Yes. I asked the oracle: "The term "real time" in this context refers to elapsed "wall clock" time, like using a stop watch. The total CPU time (user time + sys time) may be more or less than that value. Because a program may spend some time waiting and not executing at all (whether in user mode or system mode) the real time may be greater than the total CPU time. Because a program may fork children whose CPU times (both user and sys) are added to the values reported by the time command, but on a multicore system these tasks are run in parallel, the total CPU time may be greater than the real time." "...justifies the premium prices you pay for Apple hardware." Did you already order your first Mac ;-) "...Do you remember when I said a few days ago..." Indeed. Best regards, Karl Edit: Fixed typo. P.S.: I'm thinking already about episode #3 of "Threads From Hell". ŤThe Crux of the Biscuit is the Apostropheť	[reply]
Re^6: Threads From Hell #2: How To Parse A Very Huge File by BrowserUk (Patriarch) on May 26, 2015 at 10:53 UTC
Re^7: Threads From Hell #2: How To Parse A Very Huge File by karlgoethebier (Abbot) on May 26, 2015 at 19:47 UTC
Some notes below your chosen depth have not been shown here
Re^3: Threads From Hell #2: How To Parse A Very Huge File by marioroy (Prior) on May 24, 2015 at 12:57 UTC
MCE::Grep is not the tool for this. Calling a code block (once per line) is the reason for the overhead. The testing was done on a CentOS VM. However, the MCE::Loop example showed a way that allows Perl to run faster. Surely, one is not likely to force a character count. I was mainly comparing the wc binary against the Perl script.	[reply]
Re^4: Threads From Hell #2: How To Parse A Very Huge File by BrowserUk (Patriarch) on May 24, 2015 at 14:22 UTC
the MCE::Loop exampl I don't see any mention MCE::Loop in either of your posts? The testing was done on a CentOS VM I see. And on what hardware that allows you to read 2GiB at 16Gbits/second? With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked	[reply]
Re^5: Threads From Hell #2: How To Parse A Very Huge File by marioroy (Prior) on May 24, 2015 at 16:13 UTC
The testing was done on a late 2013 MacBook Pro model (Haswell Core i7) at 2.6 GHz with 1600 MHz memory. Am running Parallels Desktop 9.0. The grep/wc commands and Perl scripts read the file likely residing in OS level file cache from repeated testing.	[reply]
Re^6: Threads From Hell #2: How To Parse A Very Huge File by BrowserUk (Patriarch) on May 24, 2015 at 17:39 UTC
Re^7: Threads From Hell #2: How To Parse A Very Huge File by marioroy (Prior) on May 24, 2015 at 19:32 UTC
Some notes below your chosen depth have not been shown here