in reply to Fun with threads

In addition to the good advice already posted, you may like to consider a serious remodel of your process. Rather than joining your threads to retrieve their result, you could setup a 'return queue'. Thus you start each worker thread - passing it $somedata and a reference to your return queue such as made by threads::shared::queue. Detach the thread, and forget you ever started it. In your worker sub, ->enqueue your processed return data , and exit the thread.

I have a snippet that demonstrates simple use of queues. The networking stuff is pretty poor and not related to your problem so best to ignore that.

This way you have a simple accessor to finished data in your main thread, by ->dequeue 'ing from your queue object. Once the detached worker threads enqueue their result , they can die a natural death (exit). Which should be reflected in the number that threads->list returns. As @threads_running falls below your 'apparent' optimal thread count - top up the pool by starting another thread

Of course I'm going to run off and try this now, but methinks the theory is ok



Good Luck!

Replies are listed 'Best First'.
Re: Re: Fun with threads
by znu (Acolyte) on Dec 20, 2002 at 03:46 UTC
    Thanks, but does it work for you? The queuing is a good idea and i will implement it at some stage, but i don't think that's my problem at the moment. Essentially, I think i'm doing the same thing as you suggested except for the fact that even when I run a completely stripped down version of the script, i.e each thread does no data proccessing whatsoever, just prints to the screen i get the same result. Furthermore, if I change it so that each thread detaches itself instead of joining itself, I get a funny phenomenon where after around 1500 threads are launched the script dies printing "Killed" to the screen :-/
[reply]

      I have to admit to being v.v.confused by what you're aiming to achieve here. You pass data to the threads from getData() , checking to see that $somedata is !null , at which point things should finish. Are these threads CPU intense ? chewing some furry metrics from $somedata ? or do they use $somedata to build an IO pipe like a network socket? In the latter case , 1500 threads (phew) could be munching away occasionally on their socket. 1500 threads trying to churn $somedata . . . I'd never dare -. I'm pretty sure recommendations for perl threads say that fewer==better .


      still confused please post more code. I think this is going to bug me and poor linux box all weekend.

      Prolly be useful to know what OS, CPU , perl flavours you're stirring with this program too. : )



      i can't believe its not psellchecked
[reply]
        Hey, thanks for the concern. I looked at pg's code and i must admit i'm still confused. To me it looks like it's doing what I need it to but hanging after having launched 1020 or so threads. What i'm doing is downloading small chunks of data from the internet. I'm on a fairly fast link, but if I try to do the proccess in serial, it takes ages because of waiting for http request/response. So, i'm launching b/ween 50-100 seperate threads to do the task, this way i'm finding i can make better use of my bandwidth. The plan is to have X number of workers sitting there downloading these small chunks of data, when one of them finishes, a new worker is created to take it's place. What i'm trying to do is probably not the best way to do it, but as I said, it's working nicely apart from hanging after having created 1020 threads. Thanks for your help!
[reply]

In Section Seekers of Perl Wisdom