in reply to Re^2: passing hashes between threads
in thread passing hashes between threads

Push references to already shared hashes. -- I tried this - it was slower than the deep copy

Hm. Here's my test of several methods of passing hashes between threads:

And here are the results for 10000 hashes of 100 key/value pairs:

c:\test>TQ-b -H=200 -N=10000 Unshared hashrefs: 10.857 join/split: 3.121 freeze/thaw: 0.686 Shared hashrefs: 0.265

Here are the results for 10000 hashes of 1000 key/value pairs:

c:\test>TQ-b -H=2000 -N=10000 Unshared hashrefs: 117.532 join/split: 30.482 freeze/thaw: 2.886 Shared hashrefs: 0.250

Please note not just how much faster the latter is, but that it barely changes with hashes of 10 times the size.

I still think that your design that requires hashes to be shipped from one thread to another is the wrong approach, but you've not supplied enough information to allow me to confirm or deny that.

not only a writing but maintaining the original order

Hm. This is very troublesome. Quite how you are "maintaining order" when fanning out records to multiple threads and then gathering them back together is very unclear. Nothing in your posted code, and no mechanism I am aware of of will allow you to do this.

Threads are non-deterministic. Believing you will read records back from the 'return' queue in the same order as you fed them to the 'work' queue is a very bad assumption.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.