Beefy Boxes and Bandwidth Generously Provided by pair Networks
Come for the quick hacks, stay for the epiphanies.

Re^3: shared scalar freed early (queue)

by tye (Sage)
on Feb 22, 2017 at 20:17 UTC ( #1182556=note: print w/replies, xml ) Need Help??

in reply to Re^2: shared scalar freed early
in thread shared scalar freed early

I tried making the threads persistent and read data from a queue, but that was MUCH slower.

Sounds to me like you are doing something very strange. Creating a Perl "thread" is rather expensive. Much more expensive than pulling an item from a queue. Usually much, much more expensive.

- tye        

  • Comment on Re^3: shared scalar freed early (queue)

Replies are listed 'Best First'.
Re^4: shared scalar freed early (queue)
by chris212 (Scribe) on Feb 23, 2017 at 03:16 UTC
    Please see reply to ikegami. That is not the results I get with my test script. Thanks.

      Oh, so it is "slower" because you are doing all kinds of blocking between threads (not because it is more expensive). Don't do that.

      If you need to preserve order, than create slots for the data and let the threads fill in the slots at their leisure. Way, way simpler code. I failed at trying to understand what the point of your overly complicated sample code was. Sorry.

      Don't pass thread handles through queues. Pass work items. If you only have as many threads as you need, then you don't need the semaphore at all.

      - tye        

        It will only block between threads with testa (my approach) if the next thread to output is not the first one finished. As soon as the next thread to output is finished, a thread is created for the next chunk to be processed. This can probably even be improved if I up the semaphore at the end of the worker thread instead of after the next one to output is joined.

        With testb (your approach), the semaphore is up'ed as soon as any chunk of data finishes processing allowing another chunk to be queued and processed. That should theoretically be more efficient, but it isn't. It is 5x slower. Even with ikegami's clean and elegant solutions using your approach, it is till 5x slower. I suspect it has to do with how memory is managed passing data structures to threads as opposed to making it shared in a queue.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1182556]
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others chilling in the Monastery: (3)
As of 2023-12-08 02:07 GMT
Find Nodes?
    Voting Booth?
    What's your preferred 'use VERSION' for new CPAN modules in 2023?

    Results (34 votes). Check out past polls.