The idea is to spawn max threads once :) and use proper scoping and argument passing

Re^9: Async DNS with LWP/Re^13: Async DNS with LWP, Re^4: Perl/Tk vs Win32::SerialPort (stash, dispatch, queue), Re: Challenge: Perl 5: lazy sameFringe()?, Re^5: Consumes memory then crashs
threads::Q

This code untested but I've used this pattern before 

#!/usr/bin/perl --
Main( @ARGV );
exit( 0 );
sub Main {
   ... ; ## GetOpt::Long / GetOpt::Declare ...
   UrlFile_FetchingThreads ( '/f/o/o/bar.txt', 4 );
##    UrlFile_FetchingThreads ( $filename, $maxthreads );
}

sub UrlFile_FetchingThreads {
    my( $filename, $maxthreads ) = @_;
    my @urls = GetUrls( $filename );
    my $qin  = threads::Q->new();
    my $qout = threads::Q->new();
    $qin->nq( @urls );
    for ( 1 .. $maxthreads ){
        my $tt = threads->create( \&tryHTTP, $qin, $qout );
        $tt->detach;
    }
    
    my $quitter = 0;
    while( 1 ){
        if( my $res = $qout->dq_nb ){
            doSomething( $ret );
        }
        sleep 1;
### this part not used before, completely untested, probably too compl
+ex
        if( not $qin->cq ){
            $quitter++;
        } else {
            $quitter = 0;
        }
        if( not $qout->cq ){
            if( $quitter > 5 ){
                die "FINISHED, no more urls to process, no more respon
+ses to process\n";
            }
        }
    }
}
sub tryHTTP {
    threads->detach();    ## can't join me :)
    my( $qin, $qout ) = @_;
    while( 1 ){
        my( $url ) = $qin->dq;
        ...
        my $res = $lwp->get( $url );
        $qout->nq( $res );
    }
    return;
}

[download]

Re: simple multithreading with curl
Re: Perl crashing with Parallel::ForkManager and WWW::Mechanize
Re^3: Using LWP instead of wget?
Re^3: Fast fetching of HTML response code
Re^2: Need help with Perl multi threading
LWP::Parallel::UserAgent
LWP::Concurrent
Re^10: Consumes memory then crashs Re^9: Consumes memory then crashs Re: Are there any memory-efficient web scrapers?
Your main event may be another's side-show.
Re: Perl threads to open 200 http connections
Re^3: trying to get timeout to work (easier with threads)

In reply to Re: Proper way to thread this in PERL. (mech lwp asynchronous) by Anonymous Monk
in thread Proper way to thread this in PERL. by tekio

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.