Push references to already shared hashes. -- I tried this - it was slower than the deep copy

Hm. Here's my test of several methods of passing hashes between threads:

<Reveal this spoiler or all in this thread>

And here are the results for 10000 hashes of 100 key/value pairs:

c:\test>TQ-b -H=200 -N=10000
Unshared hashrefs: 10.857
join/split: 3.121
freeze/thaw: 0.686
Shared hashrefs: 0.265

[download]

Here are the results for 10000 hashes of 1000 key/value pairs:

c:\test>TQ-b -H=2000 -N=10000
Unshared hashrefs: 117.532
join/split: 30.482
freeze/thaw: 2.886
Shared hashrefs: 0.250

[download]

Please note not just how much faster the latter is, but that it barely changes with hashes of 10 times the size.

I still think that your design that requires hashes to be shipped from one thread to another is the wrong approach, but you've not supplied enough information to allow me to confirm or deny that.

not only a writing but maintaining the original order

Hm. This is very troublesome. Quite how you are "maintaining order" when fanning out records to multiple threads and then gathering them back together is very unclear. Nothing in your posted code, and no mechanism I am aware of of will allow you to do this.

Threads are non-deterministic. Believing you will read records back from the 'return' queue in the same order as you fed them to the 'work' queue is a very bad assumption.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

In reply to Re^3: passing hashes between threads by BrowserUk
in thread passing hashes between threads by bago

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.