Re^3: Problem with ithreads

The main algorithm may be described in few words:

There are one boss thread and number of worker threads.
Boss thread creates worker threads and they works parallel.
The boss thread generate tasks for worker threads (using Thread::Queue).
Each worker thread in while loop:
- reads task parameters from Thread::Queue object;
- execute task;
- return result to the boss thread using another Thread::Queue object.
Boss thread receives result and does something with them.

So each worker thread have to execute the same task with various parameters received from boss thread for many times and return results to the boss thread (boss <=> workers).

So thread_do code may be:

sub thread_do
{
    threads->self->detach();
    my $tid = threads->self->tid();
    while (1) {
        my $url = $task_q->dequeue();
        my $ua = LWP::UserAgent->new(timeout => 3);
        my $res = $ua->request(HEAD $url);
        $result_q->enqueue("$tid;$url;" . $res->code() . ";" . $res->m
+essage() . ";");
    }
}
[download]

As I said above, I run script and it works as I expected for the first 10-15 minutes. Every worker thread do one request per 3-4 seconds and put result into $result_q. The boss thread gets results from $result_q and prints them. So, if I run script with 50 threads I get about 10-15 results per seond:

23;200;
31;200;
32;200;
30;200;
34;200;
38;200;
35;500;
21;200;
37;200;
22;500;
27;200;
50;200;
24;200;

etc..

Than something happens with almost all worker threads – they stops and I see results like this:

30;200;
7;200;
30;200;
7;200;
7;200;
7;200;
30;200;
7;200;

I don’t know what happens with other 48 threads. Why they stop? This is the problem, I’m trying to solve and if I sole this problem I’ll be able to create script I need.

The speed in first 10-20 minutes, while all threads are working, is perfect so all I need is to prevent worker threads from stopping after some time. This is the main problem.

Comment on Re^3: Problem with ithreads Download Code

Replies are listed 'Best First'.
Re^4: Problem with ithreads by BrowserUk (Patriarch) on Dec 30, 2004 at 11:28 UTC
Why they stop? Because your running too many threads. And because your code is badly structured. This is an endless loop that does nothing useful. `while (1) { my $url = $task_q->dequeue(); my $ua = LWP::UserAgent->new(timeout => 3); my $res = $ua->request(HEAD $url); $result_q->enqueue( "$tid;$url;" . $res->code() . ";" . $res->message() . ";" ); }` [download] All your code does is prove that if you write badly structured code and run gazillions of threads, you can induce non-useful behaviour. This is the problem, I’m trying to solve... The solution is simple: Don't do that! Don't run large numbers of threads. Don't run endless, pointless loops within threads. All the code you have provided this time is a snippet of the same code from your original post, which you already said was test code--which I cannot correct, because it doesn't do anything useful. Examine what is said, not who speaks. Silence betokens consent. Love the truth but pardon error.	[reply] [d/l]
Re^5: Problem with ithreads by 2NetFly (Initiate) on Dec 30, 2004 at 14:15 UTC
I have 70 000 URLs (almost all on different servers) to check every day and I have to check them as quickly as possible (max 60 min). Depending on sever on witch document is located it take from 1 to 4 seconds to make head request. So, if I want make 70 000 / 60 / 60 = 20 requests per second, I need at least 50 threads working parallel. CPU load isn’t very high because almost all the time thread is waiting for response from the server. I’ll rewrite code a bit. `sub thread_do { # while not all urls are checked while (!$DONE) { # get new url from boss thread my $url = $task_q->dequeue(); # check url my $ua = LWP::UserAgent->new(timeout => 3); my $res = $ua->request(HEAD $url); # return result to the boss thread $result_q->enqueue( "$tid;$url;" . $res->code() . ";" . $res->message() . ";" ); }` [download] $DONE is a shared variable and the boss thread set it true when all urls are checked. I used prethreading that means that I create number of threads and each thread process the same task number of times. In my example each thread makes head request until all urls are checked.	[reply] [d/l]