in reply to How to split file for threading?

This node falls below the community's minimum standard of quality and will not be displayed.

Replies are listed 'Best First'.
Re^2: How to split file for threading?
by BrowserUk (Patriarch) on Jun 23, 2015 at 21:13 UTC
    I seriously doubt that threading would do anything positive for you.

    That's the trouble with not having the skills or capability to do your own research; you're always left regurgitating someone else's knowledge, even when it is out-of-date.

    Here are the results of multi-threading a search of an 8GB file on a 4-core machine:

    [20:39:41.83] C:\test>1131634.pl -T=1 s:\1GBx8.bin lAxc Found 10 'lAxc' lines Took 207.819706917 secs cpu(162.296 15.906 0 0) [20:43:09.82] C:\test>1131634.pl -T=2 s:\1GBx8.bin lAxc Found 10 'lAxc' lines Took 169.678593159 secs cpu(164.203 18.968 0 0) [20:46:00.25] C:\test>1131634.pl -T=3 s:\1GBx8.bin lAxc Found 10 'lAxc' lines Took 156.497795820 secs cpu(164.375 18.656 0 0) [20:48:37.34] C:\test>1131634.pl -T=4 s:\1GBx8.bin lAxc Found 10 'lAxc' lines Took 127.086592913 secs cpu(161.843 19.546 0 0) [20:50:45.04] C:\test>1131634.pl -T=5 s:\1GBx8.bin lAxc Found 10 'lAxc' lines Took 115.240671158 secs cpu(161.89 19.109 0 0) [20:52:40.87] C:\test>1131634.pl -T=6 s:\1GBx8.bin lAxc Found 10 'lAxc' lines Took 129.716307163 secs cpu(161.781 21.859 0 0) [20:54:51.19] C:\test>1131634.pl -T=7 s:\1GBx8.bin lAxc Found 10 'lAxc' lines Took 167.007865906 secs cpu(162.328 22.171 0 0) [20:57:38.78] C:\test>1131634.pl -T=8 s:\1GBx8.bin lAxc Found 10 'lAxc' lines Took 179.142831087 secs cpu(164.171 25.546 0 0) 210 --- E 200 ... 165.0 l 190 ... 164.8 a 180 ... ___ 164.6 p 170 ... ___ X ... 164.4 C s 160 ... .X. --- .X. 164.2 P e 150 ... ... --- ... ... 164.0 U d 140 ... ... ... ___ ... ... 163.8 130 ... ... ... ... ... ... 163.6 S s 120 ... ... ... --- ... ... ... 163.4 e e 110 ... ... ... ... --- ... ... ... 163.2 c c 100 .X. ... ... ... ... ... ... ... 163.0 o o 90 ... ... ... ... ... ... ... ... 162.8 n n 80 ... ... ... ... ... ... ... ... 162.6 d d 70 ... ... ... ... ... ... .X. ... 162.4 s s 60 ... ... ... ... ... ... ... ... 162.2 . . 50 ... ... ... ... ... ... ... ... 162.0 40 ... ... ... .X. .X. .X. ... ... 161.8 30 ... ... ... ... ... ... ... ... 161.6 20 ... ... ... ... ... ... ... ... 161.4 10 ... ... ... ... ... ... ... ... 161.2 1 2 3 4 5 6 7 8 T H R E A D S

    With 5 threads, it almost halves the time and and even reduces the overall cpu usage slightly.

    The only question left is why do you bother to continue to play the Village Idiot™, and I think finally the answer to that conundrum is becoming self-evident.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
    I'm with torvalds on this Agile (and TDD) debunked I told'em LLVM was the way to go. But did they listen!
      I'll admit to being a bit of a dinosaur myself on matters of multi-core machines, but even without understanding how such machines actually work, I always love a good benchmark test.

      Would you be able to share some more detail about how that chart was produced?

        Would you be able to share some more detail about how that chart was produced?

        By hand. I wanted to convey my point :)

        It's a style of hand-made chart I've used since I was at college, when "graphics" meant ascii-art.

        I guess it would be an interesting exercise to write a program to generate them; but with the other options available I doubt it would see much use.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
        I'm with torvalds on this Agile (and TDD) debunked I told'em LLVM was the way to go. But did they listen!
Re^2: How to split file for threading?
by Anonymous Monk on Jun 23, 2015 at 18:39 UTC
    Dinosaurs such as yourself should check out Hadoop.
Re^2: How to split file for threading?
by karlgoethebier (Abbot) on Jul 04, 2015 at 11:40 UTC
    "I seriously doubt that threading would do anything positive for you. "

    Caveat lector: <°)))o><

    «The Crux of the Biscuit is the Apostrophe»