in reply to Re^5: Perl Thread Quitting Abnormally
in thread Perl Thread Quitting Abnormally

Hi,

Thanks for the suggestion of using a shared array for $end_ne . I was initially using a signal to kill the thread and the restart it, when I change dto just setting an array I didn't think to use the array. Will change that.

You asked me to post the command and control code:

my @hosts = keys %nes; my @hosts_ok; my $max_threads = $config->val($section,'MaxThreads'); for my $threadno (0 .. ($max_threads-1)){ if (@hosts){ my $host = shift (@hosts); $thread_ip[$threadno] = $host; $thread_result[$threadno] = 0; $thread_time[$threadno] = time(); push @threads, threads->create('get_ne_data',$threadno); $threads[@threads-1]->detach(); write_log ('MAIN','STARTING THREADS',1,'001b','Starting thread +',$threads[@threads-1]->tid(),'NE',$host); sleep(1); } } while (@hosts){ for my $threadno (0 .. (@threads-1)){ if ($threads[$threadno] and $threads[$threadno]->is_running() +){ if ($thread_result[$threadno]){ # thread as returned if ($thread_result[$threadno] == 2){ write_log ('MAIN','PROCESSING NES',2,'001c','Threa +d',$threads[$threadno]->tid(),'NE',$thread_ip[$threadno],'Finished OK +'); push @hosts_ok, $thread_ip[$threadno]; } else { write_log ('MAIN','PROCESSING NES',2,'001d','Threa +d',$threads[$threadno]->tid(),'NE',$thread_ip[$threadno],'Finished NO +K'); } $thread_time[$threadno] = time(); $thread_result[$threadno] = 0; if (@hosts){ $thread_ip[$threadno] = shift @hosts; } else { # print "\tFINISHED\n"; $thread_ip[$threadno] = 'FINISHED'; } write_log ('MAIN','PROCESSING NES',2,'001e','Thread',$ +threads[$threadno]->tid(),'is being assigned',$thread_ip[$threadno]); } elsif ($thread_time[$threadno] < (time-60) ){ write_log ('MAIN','PROCESSING NES',2,'001f','Thread',$ +threads[$threadno]->tid(),'NE',$thread_ip[$threadno],'Being killed'); # print "\t$thread_ip[$threadno] is being killed\n"; $threads[$threadno]->kill('STOP'); $thread_time[$threadno] = time(); $thread_result[$threadno] = 0; if (@hosts){ $thread_ip[$threadno] = shift @hosts; } else { # print "\tFINISHED\n"; $thread_ip[$threadno] = 'FINISHED'; } write_log ('MAIN','PROCESSING NES',2,'0020','Thread',$ +threads[$threadno]->tid(),'is being assigned',$thread_ip[$threadno]); } } else { # for some reason we don't have a thread here - possibly sto +pped due to long run time # print "\tTrying to restart thread\n"; if (@hosts){ my $host = shift (@hosts); $thread_ip[$threadno] = $host; $thread_result[$threadno] = 0; $thread_time[$threadno] = time(); $threads[$threadno] = threads->create('get_ne_data',$t +hreadno); $threads[$threadno]->detach(); write_log ('MAIN','PROCESSING NES',4,'0021','THREAD NE +EDS RESTARTING',$threads[$threadno]->tid(),'NE',$thread_ip[$threadno] +); } } } sleep(1); } write_log ('MAIN','',2,'0022','All hosts started'); my $threads_running = 1; while ($threads_running){ $threads_running = 0; for my $threadno (0 .. (@threads-1)){ if ($threads[$threadno] and $threads[$threadno]->is_running() +){ $threads_running++; if ($thread_result[$threadno]){ # thread as returned if ($thread_result[$threadno] == 2){ write_log ('MAIN','CLEARUP',2,'0022','Thread',$thr +eads[$threadno]->tid(),'NE',$thread_ip[$threadno],'Finished OK'); push @hosts_ok, $thread_ip[$threadno]; } else { write_log ('MAIN','CLEARUP',2,'0023','Thread',$thr +eads[$threadno]->tid(),'NE',$thread_ip[$threadno],'Finished NOK'); } $thread_ip[$threadno] = 'FINISHED'; $thread_result[$threadno] = 0; } elsif ($thread_time[$threadno] < (time-60) ){ write_log ('MAIN','CLEARUP',2,'0024','Thread',$threads +[$threadno]->tid(),'NE',$thread_ip[$threadno],'Being killed'); $threads[$threadno]->kill('STOP'); $thread_time[$threadno] = time(); $thread_result[$threadno] = 0; $thread_ip[$threadno] = 'FINISHED'; } } } }

Replies are listed 'Best First'.
Re^7: Perl Thread Quitting Abnormally
by BrowserUk (Patriarch) on Jul 07, 2010 at 21:27 UTC

    Your architecture means that your main thread is doing an awful lot of "busy work". Continuously looping around polling each of the threads to see if it has completed it current work item so that it can give it a new one. That means that when a given thread finishes one item it must then wait until the main thread gets around to checking its state and issue it with another host ip before it can continue doing anything useful.

    That is not a good architecture. I suspect that your main thread will be consuming a substantial amount of your cpu resources basically doing the equivalent of the perpetually asking "Are we there yet? Are we there yet?".

    A thread pool + queue architecture, such as I outlined in Re: How to create thread pool of ithreads would be more efficient and easier to reason about.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.