Re^4: Problem with ithreads

Why they stop?

Because your running too many threads.

And because your code is badly structured. This is an endless loop that does nothing useful.

    while (1) {
        my $url = $task_q->dequeue();
        my $ua = LWP::UserAgent->new(timeout => 3);
        my $res = $ua->request(HEAD $url);
        $result_q->enqueue(
            "$tid;$url;" 
            . $res->code() 
            . ";" 
            . $res->message() 
            . ";"
        );
    }
[download]

All your code does is prove that if you write badly structured code and run gazillions of threads, you can induce non-useful behaviour.

This is the problem, I’m trying to solve...

The solution is simple: Don't do that! Don't run large numbers of threads. Don't run endless, pointless loops within threads.

All the code you have provided this time is a snippet of the same code from your original post, which you already said was test code--which I cannot correct, because it doesn't do anything useful.

Examine what is said, not who speaks.

Silence betokens consent.

Love the truth but pardon error.

Comment on Re^4: Problem with ithreads Download Code

Replies are listed 'Best First'.
Re^5: Problem with ithreads by 2NetFly (Initiate) on Dec 30, 2004 at 14:15 UTC
I have 70 000 URLs (almost all on different servers) to check every day and I have to check them as quickly as possible (max 60 min). Depending on sever on witch document is located it take from 1 to 4 seconds to make head request. So, if I want make 70 000 / 60 / 60 = 20 requests per second, I need at least 50 threads working parallel. CPU load isn’t very high because almost all the time thread is waiting for response from the server. I’ll rewrite code a bit. `sub thread_do { # while not all urls are checked while (!$DONE) { # get new url from boss thread my $url = $task_q->dequeue(); # check url my $ua = LWP::UserAgent->new(timeout => 3); my $res = $ua->request(HEAD $url); # return result to the boss thread $result_q->enqueue( "$tid;$url;" . $res->code() . ";" . $res->message() . ";" ); }` [download] $DONE is a shared variable and the boss thread set it true when all urls are checked. I used prethreading that means that I create number of threads and each thread process the same task number of times. In my example each thread makes head request until all urls are checked.	[reply] [d/l]

Replies are listed 'Best First'.

Re^5: Problem with ithreads
by 2NetFly (Initiate) on Dec 30, 2004 at 14:15 UTC

sub thread_do
{
    # while not all urls are checked
    while (!$DONE) {
        # get new url from boss thread
        my $url = $task_q->dequeue();
        # check url
        my $ua = LWP::UserAgent->new(timeout => 3);
        my $res = $ua->request(HEAD $url);
        # return result to the boss thread
        $result_q->enqueue(
            "$tid;$url;" 
            . $res->code() 
            . ";" 
            . $res->message() 
            . ";"
        );
    }
[download]

[reply]
[d/l]