Elijah has asked for the wisdom of the Perl Monks concerning the following question:

I wrote a simple proxy scanner application using threads. However I can only test it on Windows as of now because my Linux server is still packed up. The issue is the main thread exits before all worker threads are joined despite the call to join at the end of the main thread. I am not sure if this is an issue with Windows or the application itself.

#!/usr/bin/perl -w use strict; use Thread; use LWP::UserAgent::ProxyAny; my $NumPerThread = 6; open(PROXY, "<", "proxies.txt") || die "No proxy list found!\n"; my @proxies = <PROXY>; close(PROXY); my ($thread, $cnt, @group); for (@proxies) { push(@group, $_); $cnt++; if ($cnt >= $NumPerThread) { $thread = new Thread \&process, @group; undef @group; undef $cnt; } } if (scalar @group) { $thread = new Thread \&process, @group; undef @group; } $thread->join; print "died with error $@\n" if ($@); print "Main Thread Exiting!\n"; sub process { my @section = @_; for (@section) { chomp(); my $ua = LWP::UserAgent::ProxyAny->new; $ua->set_proxy_by_name($_); $ua->timeout(2); my $response = $ua->get('http://www.google.com'); if ($response->is_success) { print "Connection using $_ was successfull!\n"; open(GOOD, ">>", "good_proxies.txt") || die "Unable to open good proxy list for writing: ($!)\n"; flock(GOOD, 2); print GOOD $_,"\n"; close(GOOD); }else{ print "Connection using $_ failed!\n"; } } }

Is there something I have missed here? I have tried calling join after each thread creation and this allows each thread to finish however it blocks before continueing on to the next one due to the blocking nature of join(). Using that method is just as slow as not using threads at all.

Replies are listed 'Best First'.
Re: Main thread exist before all worker threads?
by zentara (Cardinal) on Jul 07, 2005 at 18:31 UTC
    Is there something I have missed here?

    Yes, a thread cannot be joined until it gets to the end of it's code-block. So you need to work out a way, either thru the process code, or thru shared variables, to tell the thread to go to an END: label. See Tk-with-worker-threads for an example.


    I'm not really a human, but I play one on earth. flash japh
Re: Main thread exist before all worker threads?
by sgifford (Prior) on Jul 07, 2005 at 19:11 UTC
    I haven't use threads much in Perl, so please take my advice with a grain of salt. It looks like you're just joining with the last thread you created, when really you want to join with all of your threads. One way to do this would be to keep the threads in an array, then just join then sequentially when your main program is done:
    our @threads; ... if ($cnt >= $NumPerThread) { my $thread = new Thread \&process, @group; push(@threads,$thread); undef @group; undef $cnt; } ... foreach my $thread (@threads) { $thread->join; }
Re: Main thread exist before all worker threads?
by eXile (Priest) on Jul 08, 2005 at 14:06 UTC
    as a side note I'd also encourage you to enhance your script to notify abuse@<netblock-owner> once you find an open proxy (maybe using Net::Whois to find email/domain information? )