Multithreaded memory usage

bingohighway has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Multithreaded memory usage by moritz (Cardinal) on Apr 21, 2009 at 08:01 UTC
Any ideas? Two ideas actually. The first is to search for memory leaks with something like Devel::Cycle or Devel::Leak - maybe you just have a circular reference somewhere that prevents perl from destroying the objects. The second idea is to spawn a new ~~thread~~process for each task, and let the operating system clean up the memory for you.	[reply]
Re^2: Multithreaded memory usage by bingohighway (Acolyte) on Apr 21, 2009 at 08:04 UTC
Ok, I'll try that. I am assuming once a detached thread reaches the end of its code block it should automatically quit and the OS should recover that memory? (assuming it is coded correctly :-) ) Cheers	[reply]
Re^3: Multithreaded memory usage by moritz (Cardinal) on Apr 21, 2009 at 08:26 UTC
I am assuming once a detached thread reaches the end of its code block it should automatically quit and the OS should recover that memory? I'm really not an expert here, but I don't think that's the case, at least not always. The OS probably doesn't have any idea about which parts of the memory is associated to which user-level thread, and thus can't clean up. It does know about processes though, which is why I recommended them over threads.	[reply]
Re^4: Multithreaded memory usage by bingohighway (Acolyte) on Apr 21, 2009 at 08:31 UTC
Re^5: Multithreaded memory usage by moritz (Cardinal) on Apr 21, 2009 at 09:09 UTC
Some notes below your chosen depth have not been shown here
Re^2: Multithreaded memory usage by why_bird (Pilgrim) on Apr 21, 2009 at 09:45 UTC
The second idea is to spawn a new thread for each task Did you mean process rather than thread? update: p.s. hurrah for physicists (especially those forgetting everything they learnt by doing a completely unrelated job) :) ........ Those are my principles. If you don't like them I have others. -- Groucho Marx .......	[reply]
Re^3: Multithreaded memory usage by moritz (Cardinal) on Apr 21, 2009 at 10:47 UTC
yes, that's what I meant. Sorry for the confusion.	[reply]
Re^2: Multithreaded memory usage by bingohighway (Acolyte) on Apr 21, 2009 at 13:25 UTC
Okay, I have tried the for method and after about 60 forks (over 5 mins) the parent process can't fork any more processes. I have tried setting `$SIG{CHLD} = 'IGNORE'` to ignore the zombified children, but to no avail. Any ideas where to go to next? Cheers!	[reply] [d/l]
Re^3: Multithreaded memory usage by Corion (Patriarch) on Apr 21, 2009 at 14:06 UTC
On Windows, start the subprocess not via `fork()` but better via `system(1, @args)` or via `system("start @args");`. That way, it is dissociated from the main Perl program.	[reply] [d/l] [select]
Re^3: Multithreaded memory usage by moritz (Cardinal) on Apr 21, 2009 at 13:51 UTC
wait (or waitpid) for them (perhaps in `$SIG{CHLD}` to be nearly non-blocking) so that they don't become zombies? And what's the error message?	[reply] [d/l]
Re^4: Multithreaded memory usage by bingohighway (Acolyte) on Apr 21, 2009 at 14:02 UTC
Re^5: Multithreaded memory usage by moritz (Cardinal) on Apr 21, 2009 at 14:05 UTC
Re: Multithreaded memory usage by BrowserUk (Patriarch) on Apr 21, 2009 at 15:48 UTC
Forget using Perl's Windows fork emulation. It leaks like a sieve. See how you get on with this. On my system it settles down to a steady state memory usage after a few cycles: `#! perl -sw use 5.010; use strict; use threads ( stack_size => 4096 ); while( 1 ) { async( \&get_data_and_go )->detach; sleep 5; ## Go get every 5 seconds } sub get_data_and_go { my $tid = threads->tid; require LWP::Simple; ## Requiring prevent CLONE leaks. ##does some webpage stuff and exits my $bytes= length( LWP::Simple::get( 'http://www.yahoo.com' ) ); say "$tid : Got $bytes"; sleep rand 10; ## Simulated variable processing time say "$tid : done"; return 0; };` [download] Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply] [d/l]
Re^2: Multithreaded memory usage by bingohighway (Acolyte) on Apr 21, 2009 at 20:20 UTC
Superb :-) Did the trick. No memory leaks and seems to be less CPU intensive. Cheers!	[reply]
Re: Multithreaded memory usage by ikegami (Patriarch) on Apr 21, 2009 at 14:03 UTC
Try explicitly undefining the variables that hold the thread reference. You might have a circular loop caused by a function captured the thread variable.	[reply]