i thought threads are supposed to take up less memory than processes...hmm. So whats the advantage of threads in perl?

The only real advantage of using threads, is you can easily share and return values between threads, with "use shared". Otherwise forking is probably faster and more memory efficient.

---

I'm not really a human, but I play one on earth. Cogito ergo sum a bum

• How much memory is being shared between your child processes? I.e. what does 'top' report in the 'SHARED' column?
• If they are not sharing a lot of memory, do you have use LWP; and use WWW::Mechanize; 'outside' your while loop (like at the beginning of your program)? This will enable them to share those modules.

i'm on os-x and hardly know how to use top on this craptop. I checked my memory usage through activity monitor. The root says 29mb and everything else (upto 20 forks or so) say 6.5-7.5 mb real memory + 73mb virtual. So should i not bother with threads and simply optimize this forked code?

Try this and see how you get on?

On my system with only dialup bandwidth I used HEAD requests and parsed local content (32kb HTML with ~1000 links). The result is that with 10 threads, it consumes ~40MB of ram and achieves >80% cpu usage. Using more threads is pointless, even fetching only head requests. That pretty much saturates my dialup connection.

YMMV depending on the size of the downloads, complexity of the processing and what other modules you need to load. Along with the number of cpus and your bandwidth. Feedback appreciated.

#! perl -slw
use strict;
use threads;
use threads::shared;

$|++; ## Important to prevent IO overlap

our $NTHREADS ||= 10;

## Read test content from DATA 32kb html containing ~ 1000 links,
## Used for testing in conjunction with head() below.
# my $content; { local $/; $content = <DATA>; }

my $osync :shared;
my $isync :shared;

sub processEm {
    require LWP::Simple; LWP::Simple->import( 'head', 'get' );
    require HTML::LinkExtor;

    my $tid = threads->self->tid;
    warn "$tid: starting";

    while( my $url = do{ lock $isync; <STDIN> } ) {
        chomp $url;
        warn "$tid: processing $url";

##      Used for testing. A workaround my bandwidth limits 
##      and being a good netizen
#        head( $url ) or warn "Couldn't fetch $url" and next;

        my $content = get( $url ) or warn "Couldn't fetch $url" and ne
+xt;

        my $l = HTML::LinkExtor->new(
            sub{ lock $osync; print "'$_[ 2 ]'"; }
            , $url
        );
        $l->parse( $content );
    }
}

open STDIN, '<', $ARGV[ 0 ] or die $!;
my @threads = map{ threads->create( \&processEm ) } 1 .. $NTHREADS;
$_->join for @threads;

__END__
c:\test>663223.plt -NTHREADS=10 urls.txt >output.txt
[download]

You may want to check out POE::Component::Client::UserAgent. If you've never worked with POE before, it's a great opportunity to learn a bit about this framework.

G'luck,
Kirill