I'm no threads expert but one thing I see problematic in your code is that you use sleep as a crude delimiter of threads in the hope that 5 threads per 5 seconds is not too much. Much better would be to check for number of threads with scalar(threads->list()) and sleep as long as this number is at maximum. Only when that number goes down another thread should be generated

Something like this:

while (threads->list()<$threadsmax) {
  sleep(1); # or threads->yield; but I'm not sure what happens if you 
+call this in the main thread 
}
print ".";
[download]

I added the print for debugging purposes. If you see that no new "." get printed after a while it would mean no threads finish.

Thank you! I will give that a try!

I would strongly suggest avoiding spawning new threads in a 'while' loop. That just causes you grief. Each time you 'create' a thread, it copies your program state, which becomes a particular problem when you're loading lots of modules. Instead, I would advocate a 'worker thread' approach, and use Thread::Queue to 'feed' them.

my $url_q = Thread::Queue -> new();

sub http_fetch_thread
{
  my $ua = LWP::UserAgent -> new(); 
  $ua -> timeout ( 10 );
  while ( my $item = $url_q -> dequeue() )
  {
    my $url = "http://www." . $item . ".com";
    <... fetch stuff ... >
    
  }
}
[download]

Spawn a defined number of these threads (you look like you're trying to keep to 5?) and then just feed the contents of your file, into '$url_q'.

for ( 1..$thrCount )
{
  threads -> create ( \&http_fetch_thread );
}

open ( my $contents, "<", $contents_file_name ) or die $!;
$url_q -> enqueue ( <$contents> ); 
$url_q -> end;
close ( $contents ); 

#wait for completion
foreach my $thr ( threads -> list() )
{
  $thr -> join(); #not capturing return code, we started in a void con
+text. 
}

print "At program completion, total count was ", $totalCount,"\n";
[download]

So rather than starting a new thread every line of your file, which is instantiating a new 'useragent' object, you'll create a number equal to the number of threads defined - and then run through the list as fast as they can, and probably won't chew up your memory anything like as badly (and because you're not creating/destroying useragents and threads, you'll probably find it runs a lot faster too).

If you want a running total of 'totalCount' you can either print it from within the thread, or instead of doing the 'foreach/join' loop, do a 'while' loop:

while ( threads -> list )
{
  foreach my $thread ( threads -> list ( threads::joinable ) )
  {
    $thread -> join();
  }
  print $totalCount,"\n";
  sleep 5;
}
[download]

Edit: More generally I'd suggest:

• 3 argument 'opens' are nicer, especially when threading. (When is 'CONTENT' in scope?)
• Detaching a thread can mean your program completing without the thread finishing. That can create anomalous results, so I'd suggest avoiding it generally
• Parsing '@ARGV' by hand is a good way to introduce bugs. Look at GetOpt::Std for anything more than very trivial cases.

These are excellent replies! Thank you guys so much! :)

The idea is to spawn max threads once :) and use proper scoping and argument passing

Re^9: Async DNS with LWP/Re^13: Async DNS with LWP, Re^4: Perl/Tk vs Win32::SerialPort (stash, dispatch, queue), Re: Challenge: Perl 5: lazy sameFringe()?, Re^5: Consumes memory then crashs
threads::Q

This code untested but I've used this pattern before

#!/usr/bin/perl --
Main( @ARGV );
exit( 0 );
sub Main {
   ... ; ## GetOpt::Long / GetOpt::Declare ...
   UrlFile_FetchingThreads ( '/f/o/o/bar.txt', 4 );
##    UrlFile_FetchingThreads ( $filename, $maxthreads );
}

sub UrlFile_FetchingThreads {
    my( $filename, $maxthreads ) = @_;
    my @urls = GetUrls( $filename );
    my $qin  = threads::Q->new();
    my $qout = threads::Q->new();
    $qin->nq( @urls );
    for ( 1 .. $maxthreads ){
        my $tt = threads->create( \&tryHTTP, $qin, $qout );
        $tt->detach;
    }
    
    my $quitter = 0;
    while( 1 ){
        if( my $res = $qout->dq_nb ){
            doSomething( $ret );
        }
        sleep 1;
### this part not used before, completely untested, probably too compl
+ex
        if( not $qin->cq ){
            $quitter++;
        } else {
            $quitter = 0;
        }
        if( not $qout->cq ){
            if( $quitter > 5 ){
                die "FINISHED, no more urls to process, no more respon
+ses to process\n";
            }
        }
    }
}
sub tryHTTP {
    threads->detach();    ## can't join me :)
    my( $qin, $qout ) = @_;
    while( 1 ){
        my( $url ) = $qin->dq;
        ...
        my $res = $lwp->get( $url );
        $qout->nq( $res );
    }
    return;
}
[download]

There’s definitely a “right way” and a “wrong way” to use threading/processes, IMHO, no matter what programming language(s) you are using to do the job.   A thread or process is a worker, not a unit of work.   The best example of concurrency in action is ... a fast-food restaurant.