Re^3: Threads From Hell #2: How To Parse A Very Huge File

Replies are listed 'Best First'.
Re^4: Threads From Hell #2: How To Parse A Very Huge File by BrowserUk (Patriarch) on May 24, 2015 at 14:22 UTC
the MCE::Loop exampl I don't see any mention MCE::Loop in either of your posts? The testing was done on a CentOS VM I see. And on what hardware that allows you to read 2GiB at 16Gbits/second? With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked	[reply]
Re^5: Threads From Hell #2: How To Parse A Very Huge File by marioroy (Prior) on May 24, 2015 at 16:13 UTC
The testing was done on a late 2013 MacBook Pro model (Haswell Core i7) at 2.6 GHz with 1600 MHz memory. Am running Parallels Desktop 9.0. The grep/wc commands and Perl scripts read the file likely residing in OS level file cache from repeated testing.	[reply]
Re^6: Threads From Hell #2: How To Parse A Very Huge File by BrowserUk (Patriarch) on May 24, 2015 at 17:39 UTC
the file likely residing in OS level file cache from repeated testing. Indeed. That's why I used a 10GB file for my testing. I've only got 8GB of ram, so there's no way for the file to get read from cache on subsequent tests. In the real world where the file being searched is coming off a disk or SSD, there is no benefit to multi-tasking grep. Even in the extremely rare case of grepping the same file multiple times, although your numbers: <Reveal this spoiler or all in this thread> show the a reduction in elapsed time, the cpu usage is actually 2.527/2.127 *100 = 19% higher. If the user is (for the sake of a term) an end-user, who types the command and hits enter, the 1 second or so saving is probably less time than it took him to decide what to type and type it; and certainly less than he will take to decide what to do with the information it produces. On the other hand, if the user is a sysadmin guy trying to balance the needs of many processes across a farm of servers, using that extra 19% of cpu resource is probably a bad thing. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked	[reply] [d/l]
Re^7: Threads From Hell #2: How To Parse A Very Huge File by marioroy (Prior) on May 24, 2015 at 19:32 UTC
Re^8: Threads From Hell #2: How To Parse A Very Huge File by BrowserUk (Patriarch) on May 24, 2015 at 20:15 UTC
Some notes below your chosen depth have not been shown here


P is for Practical
	PerlMonks