http://qs1969.pair.com?node_id=1069421


in reply to Re^4: Threaded Code Not Faster Than Non-Threaded -- Why?
in thread Threaded Code Not Faster Than Non-Threaded -- Why?

It seems like it will take a bit of experimentation to figure out how long to sleep and how deep to keep the queue.

These are not as critical as you might first think.

  1. sleep: A thread that calls $Q->pending once every millisecond will consume so few cycles that it barely registers on taskmanager/top.

    Try this:

    perl -MThread::Queue -E"$Q=new Thread::Queue; $Q->pending while Win32: +:Sleep( 1 )"

    On my system, that shows up in Process Manager as 0.0%.

    So, running it at a frequency of once every timeslice -- ~10 milliseconds -- will consume negligible cpu whilst ensuring that the queue is (re)populated in a timely manner.

  2. Queue Length: A queue, shared between 16 threads, populated with a million longish paths, will consume less than 200MB.

    Try:

    perl -Mthreads -MThread::Queue -E"async{ sleep }->detach for 1 .. 16; +$Q=new Thread::Queue; $Q->enqueue( qq[/a/longish/path/and/filename$_] + ) for 1 .. 1e6; sleep;"

    Which given a typical modern system with 4-16 GB is nothing to concern ourselves with; and is at least 3 order of magnitude greater than I'd recommend as a starting point (100 for 16 threads).

For IO-bound process like this, tuning either or both variables over quiet a large range of values will have little or no effect on overall throughput.


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^6: Threaded Code Not Faster Than Non-Threaded -- Why?
by Tommy (Chaplain) on Jan 06, 2014 at 03:15 UTC

    The results are in. I forked and refactored the code to follow the pattern you laid out (BrowserUk). The threaded part of the code is over 2X faster now! Thanks again!

    Tommy
    A mistake can be valuable or costly, depending on how faithfully you pursue correction

      Could you post (a link to) your latest code?


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

        It isn't nearly finished and is undergoing a transformation since I typed my response, but at the time of this writing you can find the threading code here

        One of the first things you'll notice is that I didn't implement a "guard" or "doorman". I'm just stuffing everything in the queue for now. Even on directory trees larger than 200GB it barely takes more than 200MB RAM.

        Tommy
        A mistake can be valuable or costly, depending on how faithfully you pursue correction

        It isn't nearly finished and is undergoing a transformation since I typed my response, but at the time of this writing you can find the threading code here

        One of the first things you'll notice is that I didn't implement a "guard" or "doorman". I'm just stuffing everything in the queue for now. Even on directory trees larger than 200GB it barely takes more than 200MB RAM.

        Tommy
        A mistake can be valuable or costly, depending on how faithfully you pursue correction