I'm really having a difficult time finding a good example of how to do this. All the posts I find using Google, the poster has asked the question and just been told the they, and I are threading the application is incorrect.

But nobody has given an example of a proper way to do this. Also, most of the PERL threading tutorials I find are just printing out something from a thread, in a "hello world" type example.

As a random project, I have created an application that reads a list of words from a text file. It then uses LWP or IO::Socket to check and see if they are a valid domain on the Interwebz. (yes, I could use whois, but that is not the point).

After about 5 minutes, all my swap space is eaten up and the o/s comes to a screeching halt. I understand why. All my threads are taking up resources not being freed. I am doing something wrong.

What I want to do is read the test file; use a variable to define how many threads can be active at one time. Get back the memory from those threads, then do $n more threads at a time.

Below is my failed attempt: (i understand the code below probably sucks in all sorts of ways. Sorry if anyone is offended)

#!/usr/bin/perl
use LWP::UserAgent;
use threads;
use threads::shared;
our $totalCount :shared;
$totalCount = 1;
$count = 1;
$k=1;
$thrCount = 5;
$timeout = 5;
$start = localtime();
 for($i=0; $i<=$#ARGV; $i++) {
    if($ARGV[$i] eq "\-t"){
    $thrCount = $ARGV[$i+1];
    }
    elsif($ARGV[$i] eq "\-i") {
    $infile = $ARGV[$i+ 1];
        }
    }
open(CONTENTS, "<$infile) or die("Could not open file!");
while($input = <CONTENTS>) {
    chomp $input;
    print "trying..... " . $input . "\n";
    print "    Total Count: " . $totalCount . "  Count: " . $count . "
+\n";
    $thr = threads->new(\&tryHTTP,$input);
    $thr->detach();
    $count++;
    $k++;
    if($k % $thrCount == 0) {
        sleep($timeout);
        $k=1;
    }
    
}
    
close(CONTENTS);

$end = localtime();
print $start . "\n";
print $end . "\n";
print $totalCount . "/ " . $count . "\n";

sub tryHTTP {
    $ua = LWP::UserAgent->new;
    
    $ua->timeout((10));
    $url = "http://www." . $_[0] . ".com";
    $response = $ua->get($url);
    if($response->is_success) {
        $totalCount++;
        $response = "";
    }
}

[download]
P.S. Merry Christmas!

In reply to Proper way to thread this in PERL. by tekio

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.