Any ideas?

Two ideas actually. The first is to search for memory leaks with something like Devel::Cycle or Devel::Leak - maybe you just have a circular reference somewhere that prevents perl from destroying the objects.

The second idea is to spawn a new threadprocess for each task, and let the operating system clean up the memory for you.

Ok, I'll try that. I am assuming once a detached thread reaches the end of its code block it should automatically quit and the OS should recover that memory? (assuming it is coded correctly :-) )

Cheers

I am assuming once a detached thread reaches the end of its code block it should automatically quit and the OS should recover that memory?

I'm really not an expert here, but I don't think that's the case, at least not always. The OS probably doesn't have any idea about which parts of the memory is associated to which user-level thread, and thus can't clean up. It does know about processes though, which is why I recommended them over threads.

The second idea is to spawn a new thread for each task

Did you mean process rather than thread?

update: p.s. hurrah for physicists (especially those forgetting everything they learnt by doing a completely unrelated job) :)

........
Those are my principles. If you don't like them I have others.
-- Groucho Marx
.......

yes, that's what I meant. Sorry for the confusion.

Okay,

I have tried the for method and after about 60 forks (over 5 mins) the parent process can't fork any more processes. I have tried setting $SIG{CHLD} = 'IGNORE' to ignore the zombified children, but to no avail.

Any ideas where to go to next?

Cheers!

On Windows, start the subprocess not via fork() but better via system(1, @args) or via system("start @args");. That way, it is dissociated from the main Perl program.

wait (or waitpid) for them (perhaps in $SIG{CHLD} to be nearly non-blocking) so that they don't become zombies? And what's the error message?

Forget using Perl's Windows fork emulation. It leaks like a sieve.

See how you get on with this. On my system it settles down to a steady state memory usage after a few cycles:

#! perl -sw
use 5.010;
use strict;
use threads ( stack_size => 4096 );

while( 1 ) {
    async( \&get_data_and_go )->detach;
    sleep 5; ## Go get every 5 seconds
}

sub get_data_and_go {
    my $tid = threads->tid;
    require LWP::Simple; ## Requiring prevent CLONE leaks.
    
    ##does some webpage stuff and exits
    my $bytes= length(  LWP::Simple::get( 'http://www.yahoo.com' ) );
    say "$tid : Got $bytes";

    sleep rand 10; ## Simulated variable processing time
    say "$tid : done";
    return 0;
};
[download]

Superb :-) Did the trick. No memory leaks and seems to be less CPU intensive.

Cheers!