Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re^2: Threads From Hell #2: How To Parse A Very Huge File

by BrowserUk (Patriarch)
on May 24, 2015 at 07:56 UTC ( [id://1127564]=note: print w/replies, xml ) Need Help??


in reply to Re: Threads From Hell #2: How To Parse A Very Huge File
in thread Threads From Hell #2: How To Search A Very Huge File [SOLVED]

And then there's:

$ time grep karl very_huge.file nose cuke karl real: 2.127s == user: 1.845s + sys: 0.283s ## CLOSE ENOUGH! $ time ./MCE-1.608/bin/mce_grep karl very_huge.file nose cuke karl real: 1.061s != user: 2.176s + sys: 1.616s ## NOT EVEN CLOSE to: 3. +792s $ time ./MCE-1.608/examples/egrep.pl karl very_huge.file nose cuke karl real: 0.690s != user: 2.165s + sys: 0.362s ## NOR IS THIS CLOSE to: + 2.527s

Looks dodgy to me.

And then the claim that:

The following code snippet parses the 2 GiB file in 1 second.

Let's examine that. The code: slurps the entire 2GiB file into memory and then starts 4 workers that each get a reference to (1/4 of???) the slurped data and then:

  1. Run a regex against their chunk of the file to see if it contains the sought string;
  2. And if it does, open their referenced chunk of the file as a memory file, and then re-run the regex on a line-by-line basis in order to get the whole lines that contain the sought string;
  3. And then they return those lines to the parent process (via a pipe??) for it to print out.

So the regex is run against 1.25 times as much data as is contained in the file, and takes "less than one second"; which is less than the 2.127 seconds the real grep takes, despite that it only processes the file's data once?

Have you heard the song: Funny numbers?


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

Replies are listed 'Best First'.
Re^3: Threads From Hell #2: How To Parse A Very Huge File
by marioroy (Prior) on May 24, 2015 at 13:27 UTC

    Chunking is applied following a bank-teller model. Thus, MCE does not read the entire file into memory.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1127564]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (7)
As of 2024-04-23 19:21 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found