I would strongly suggest avoiding spawning new threads in a 'while' loop. That just causes you grief. Each time you 'create' a thread, it copies your program state, which becomes a particular problem when you're loading lots of modules. Instead, I would advocate a 'worker thread' approach, and use Thread::Queue to 'feed' them.

my $url_q = Thread::Queue -> new();

sub http_fetch_thread
{
  my $ua = LWP::UserAgent -> new(); 
  $ua -> timeout ( 10 );
  while ( my $item = $url_q -> dequeue() )
  {
    my $url = "http://www." . $item . ".com";
    <... fetch stuff ... >
    
  }
}
[download]

Spawn a defined number of these threads (you look like you're trying to keep to 5?) and then just feed the contents of your file, into '$url_q'.

for ( 1..$thrCount )
{
  threads -> create ( \&http_fetch_thread );
}

open ( my $contents, "<", $contents_file_name ) or die $!;
$url_q -> enqueue ( <$contents> ); 
$url_q -> end;
close ( $contents ); 

#wait for completion
foreach my $thr ( threads -> list() )
{
  $thr -> join(); #not capturing return code, we started in a void con
+text. 
}

print "At program completion, total count was ", $totalCount,"\n";
[download]

So rather than starting a new thread every line of your file, which is instantiating a new 'useragent' object, you'll create a number equal to the number of threads defined - and then run through the list as fast as they can, and probably won't chew up your memory anything like as badly (and because you're not creating/destroying useragents and threads, you'll probably find it runs a lot faster too).

If you want a running total of 'totalCount' you can either print it from within the thread, or instead of doing the 'foreach/join' loop, do a 'while' loop:

while ( threads -> list )
{
  foreach my $thread ( threads -> list ( threads::joinable ) )
  {
    $thread -> join();
  }
  print $totalCount,"\n";
  sleep 5;
}
[download]

Edit: More generally I'd suggest:

• 3 argument 'opens' are nicer, especially when threading. (When is 'CONTENT' in scope?)
• Detaching a thread can mean your program completing without the thread finishing. That can create anomalous results, so I'd suggest avoiding it generally
• Parsing '@ARGV' by hand is a good way to introduce bugs. Look at GetOpt::Std for anything more than very trivial cases.

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.