Beefy Boxes and Bandwidth Generously Provided by pair Networks
Come for the quick hacks, stay for the epiphanies.
 
PerlMonks  

Re^2: Threads From Hell #2: How To Parse A Very Huge File

by BrowserUk (Patriarch)
on May 24, 2015 at 07:30 UTC ( [id://1127560]=note: print w/replies, xml ) Need Help??


in reply to Re: Threads From Hell #2: How To Parse A Very Huge File
in thread Threads From Hell #2: How To Search A Very Huge File [SOLVED]

Okay. From your post:

$ time wc -l very_huge.file 100000001 very_huge.file real 0m0.781s user 0m0.521s sys 0m0.260s

So, user code time: 0.521 + system code time: 0.260 = real time: 0.781. Makes sense!

But then:

$ time ./MCE-1.608/examples/wc.pl -l very_huge.file 100000001 very_huge.file real 0m0.612s user 0m0.633s sys 0m1.522s

User code time: 0.633 + system code time: 1.522 = real time: 2.155s != 0.612s doesn't!

The other examples seem to indicate -- although the "timing information" is equally futzed -- that if you force the real wc to run more slowly, then your fake wc runs less, more slowly?

Is the lesson of your post that if you force wc to count characters -- which you can determine instantly from ls/dir -- and words -- which nobody is ever interested in; whilst also forcing it to use an 'unnatural' collating sequence that is equally of no interest, then your MCE::Grep can do what nobody wants done, less slowly than the system (which?) provided binary does, what nobody wants done?

Great advert.


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

Replies are listed 'Best First'.
Re^3: Threads From Hell #2: How To Parse A Very Huge File
by karlgoethebier (Abbot) on May 25, 2015 at 00:07 UTC
    "...User code time... + system code time... = real time..."

    Yes:

    From the docs:

    "The time utility executes and times the specified utility. After the utility finishes, time writes to the standard error stream, (in seconds): the total time elapsed, the time used to execute the utility process and the time consumed by system overhead."

    Some observations:

    karls-mac-mini:monks karl$ ls -hl very_huge10GB.file -rw-r--r-- 1 karl karl 10G 25 Mai 00:53 very_huge10GB.file karls-mac-mini:monks karl$ time grep karl very_huge10GB.file nose cuke karl nose cuke karl nose cuke karl nose cuke karl nose cuke karl real 2m42.126s user 0m20.437s sys 0m5.645s

     

    karls-mac-mini:monks karl$ ./mce_loop.pl nose cuke karl nose cuke karl nose cuke karl nose cuke karl nose cuke karl Took 150.555 seconds

     

    #!/usr/bin/env perl use Time::HiRes qw( time ); use feature qw(say); my $start = time; say qx (grep karl very_huge10GB.file); printf "Took %.3f seconds\n", time - $start; __END__ karls-mac-mini:monks karl$ ./wrap.pl nose cuke karl nose cuke karl nose cuke karl nose cuke karl nose cuke karl Took 157.265 seconds

    For the grep example 60+60+42=162 which is 2m42s. But user+sys (20+5) is 0m25s. What do i miss?

    Perhaps it's too late tonight. Or too early in the morning?

    Best regards, Karl

    «The Crux of the Biscuit is the Apostrophe»

      For the grep example 60+60+42=162 which is 2m42s. But user+sys (20+5) is 0m25s. What do i miss?

      The wait time is not computed in the time output. This demonstration is IO bound. Basically, CPUs idle when waiting for IO.

      For the grep example 60+60+42=162 which is 2m42s. But user+sys (20+5) is 0m25s. What do i miss?

      My interpretation of those numbers is that difference of 137 seconds is the elapsed time when the processor is doing nothing (for this process) because the process is in the scheduler queue in an IO wait state, waiting for the disk.

      That's when the opportunities for saving through multiprocessing simply don't exist. (As I detailed in my first reply above.)

      It's also where marioroy's examples defy my analysis; because his system has the fastest IO rate I've ever seen. When my customer was planning to install PCIe SSDs in his server farm -- which seems like last week, but looking back was over 20 months ago -- the fastest commodity priced (he needed lots of them) cards available were Fusion-IO ioXtreme Pro 4-lane cards which were capable of something like 2.1Gbits/s (remember divide by at least 8 for GBytes/s). I guess that (somewhat) justifies the premium prices you pay for Apple hardware.

      Interpreting those real/user/sys numbers gets further complicated when the elapsed time is less than the combined user+sys time, which comes about when multiple cores are processing concurrently, thus the process is racking up 4 (number of cores) seconds of cpu for every one second of elapsed time.

      Then the waters get really muddy, when the IO waits on 4 cores, start balancing out the 4 seconds/second of cpu accumulations when there is processing to be done, and you end up with numbers that make it look like your sometimes-IO-bound/sometimes-CPU-bound process is doing 1 for 1, cpu to real seconds; but actually requires the use of 4 cores to achieve it.

      Do you remember when I said a few days ago that it was very hard to draw generic conclusions about multi-threading ... :)


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
      In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
        "...the elapsed time when the processor is doing nothing..."

        Yes. I asked the oracle:

        "The term "real time" in this context refers to elapsed "wall clock" time, like using a stop watch. The total CPU time (user time + sys time) may be more or less than that value. Because a program may spend some time waiting and not executing at all (whether in user mode or system mode) the real time may be greater than the total CPU time. Because a program may fork children whose CPU times (both user and sys) are added to the values reported by the time command, but on a multicore system these tasks are run in parallel, the total CPU time may be greater than the real time."
        "...justifies the premium prices you pay for Apple hardware."

        Did you already order your first Mac ;-)

        "...Do you remember when I said a few days ago..."

        Indeed.

        Best regards, Karl

        Edit: Fixed typo.

        P.S.: I'm thinking already about episode #3 of "Threads From Hell".

        «The Crux of the Biscuit is the Apostrophe»

Re^3: Threads From Hell #2: How To Parse A Very Huge File
by marioroy (Prior) on May 24, 2015 at 12:57 UTC

    MCE::Grep is not the tool for this. Calling a code block (once per line) is the reason for the overhead. The testing was done on a CentOS VM. However, the MCE::Loop example showed a way that allows Perl to run faster.

    Surely, one is not likely to force a character count. I was mainly comparing the wc binary against the Perl script.

      the MCE::Loop exampl

      I don't see any mention MCE::Loop in either of your posts?

      The testing was done on a CentOS VM

      I see. And on what hardware that allows you to read 2GiB at 16Gbits/second?


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
      In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

        The testing was done on a late 2013 MacBook Pro model (Haswell Core i7) at 2.6 GHz with 1600 MHz memory. Am running Parallels Desktop 9.0. The grep/wc commands and Perl scripts read the file likely residing in OS level file cache from repeated testing.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1127560]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others sharing their wisdom with the Monastery: (4)
As of 2024-03-28 22:41 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found