Which version of threads are you using? Upgrading to 0.71 seems to avoid memory leaks that the combination of 5.10 and some earlier versions (eg. 0.67) exhibited.
There is no way that you should using 100% cpu with 10 threads performing IO. This seems to be a problem with Parallel::ForkManager on 5.10. You can do pretty much exactly the same thing as above, but using threads, like this:
#! perl -slw use threads; use threads::shared; use LWP::UserAgent; use HTTP::Request; my $semStdout :shared; my $running :shared = 0; open(LIST,"urls.txt"); while ( my $tld = <LIST> ) { chomp $tld; Win32::Sleep( 100 ) while do{ lock $running; $running >= 10 }; async{ { lock $running; ++$running; } my $url = "http://$tld/"; my $ua = new LWP::UserAgent; $ua->timeout(5); $ua->agent("Mozilla/6.0"); my $req = HTTP::Request->new('GET',$url); my $res = $ua->request($req); my $content = $res->content; my $status = $content =~ /OK/i ? 'ack' : 'nak'; { lock $semStdout; printf "(%3d)$tld: %s\n", threads->self->tid, $status; } { lock $running; --$running; } }->detach; } close(LIST);
Memory usage seems to be stable and cpu usage < 10% for 10 threads.
There are better, lower resource intensive ways of using threads, but it does have the virtue of being very close to the P::FM way of operating which you might consider a bonus.
In reply to Re: Parallel::ForkManager (high cpu and a lot of memory)
by BrowserUk
in thread Parallel::ForkManager (high cpu and a lot of memory)
by marto9
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |