in reply to Parallel::ForkManager (high cpu and a lot of memory)

ActiveState Perl 5.10 on Win32 now bundles threads, threads::shared, and Thread::Queue -- which will do what you want... Here's sample code using threads and Thread::Queue that does what yours does... in a few more lines though.

Dewey

use strict; use warnings; use LWP::UserAgent; use HTTP::Request; use threads; use Thread::Queue; my $MAXTHREADS = 2; #10; sub getURL (); sub endThreads(); my $q = Thread::Queue->new(); my @threadlist = (); foreach my $y (1..$MAXTHREADS) { my $thr = threads->create('getURL'); push(@threadlist,$thr); } open( my $LIST, '<', 'lan.txt' ); while (my $line = <$LIST>) { chomp($line); $q->enqueue($line); } close($LIST); endThreads(); exit(0); sub getURL() { my $ua = new LWP::UserAgent; $ua->timeout(2); $ua->agent("Mozilla/6.0"); my $tid = threads->tid(); print "started thread $tid\n"; while (my $line = $q->dequeue()) { last if lc(substr($line,0,4)) eq 'exit'; my $url = "http://$line"; #/index.php"; my $req = HTTP::Request->new('GET',$url); my $res = $ua->request($req); my $content = $res->content; if ($content =~ /ok/) { print "<$tid>retrieved content from $line\n"; } else { print "<$tid>could not retrieve content from $line\n"; } } } sub endThreads() { foreach my $y (1..$MAXTHREADS) { $q->enqueue('EXIT'); } while (scalar(@threadlist)) { my @newthreadlist = (); foreach my $thr (@threadlist) { if ($thr->is_joinable()) { my $tid = $thr->tid(); $thr->join(); print "joined thread $tid\n"; } else { push(@newthreadlist,$thr); } } @threadlist = @newthreadlist; sleep(1) if scalar(@threadlist); } }

Replies are listed 'Best First'.
Re^2: Parallel::ForkManager (high cpu and a lot of memory)
by marto9 (Beadle) on Oct 08, 2008 at 10:36 UTC
    This code is longer, but it works. Thx!
    Apparantly Parallel::ForkManager doesn't work good with Win32.
    The CPU isn't 100% anymore, but the memory usage is still quite big. When I'm using for example 20 threads the process uses 120mb ram and it stays 120mb.

    My question is: is it normal that it uses that much memory?

      Sounds right. Perl variables are copied into every thread, so it sounds like a reasonable amount for 20 threads.