Hi Monks, first post, and I'm a bit of a novice at this stuff. Basically I want some code that will open a list of sites (in parallel), and search with those pages to find something. I have the code for that, but the problem with it (from what I can tell) is that it works a bit too good, and yet not good enough. It spawns a new thread for each site it opens up, which is cool and all, except for the fact that there can sometimes be a LOT of sites. This can bog down the server pretty quickly. I was thinking that one way I can fix it is if I can somehow limit the number of threads that run simultaneously to like 10 or 20. I sorta have to work within the modules below as well. LWP::Parallel (or whatever it's called) won't install as a module on my system, and also, LWP::UserAgent doesn't like to be used in threads, so HTTP::Lite seems like the best option ATM. One direction I was thinking was to implement some queuing, but I honestly don't know how to make that work with this code, but any suggestions are greatly appreciated. So here's the code: 
use HTTP::Lite;
use threads;
my $a = 0;
my $i = 0;

sub request
{
    print "trying to open up $address\n", scalar(localtime), "\n";
    $http = new HTTP::Lite;
    $req = $http->request("$address") 
        or die "Unable to get document: $!";
    $i++;
    my $content = $http->body();
    print "   Done with request for $address\n";
    my $search_variable = "test";
    if ( $content =~ m/$search_variable/) 
        {
            $a++ while ($content =~ m/$search_variable/g);
        }
     print "       Finished searching $address for $search_variable", 
+scalar(localtime), "\n";
}

my @addresses = (
'http://www.website1.com',
'http://www.website2.com',
.
.
.
'http://www.website50.com'
);



                 #my $URLResult = $Result->Url;
my @threads;
foreach $address(@addresses) {
    my $thread = threads->create(\&request);
    push(@threads, $thread);
}

}
foreach (@threads)
{
$_->join();
}

[download]

In reply to Threaded Web requests by schnibitz

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.