Re^9: memory leaks with threads

It's no great surprise that it's not the system apis that are at fault, but the perl source code on top of them.

I'll however wait a bit before submitting a bug report, maybe someone else knows a solution.

Okay, but if the last version of code I posted leaks that badly on 5.9.5, it's most unlikely that there is anything that can be done at Perl level (ie. in your code) to 'fix' the problem. You're going to have to wait at least until dave_the_m and co. at p5p track down and fix whatever is wrong. That's why I suggested the perlbug to ensure that a) it comes to their attention; b) they get all the relevant information they need.

Looking at the problem from a different angle. As you saw by comparing the speed and memory usage of pthreads at the C level, and ithreads at the perl level, ithreads are very heavy by comparison. That's easily explained by the benefits they give you--automatic memory management, explicitly shared, rather than implicitly shared variable spaces, and everything else that is Perl over C. And the work that is involved in achieving those--basically, starting a new interpreter and cloning the current environment each time you start a thread.

What all that is building up to, is that spawning threads for small amounts of work and then throwing them away--whilst very effective in C--is not the best way of tackling a threading problem using ithreads and perl. A better way is to use a pool of threads and re-use them.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

Comment on Re^9: memory leaks with threads

Replies are listed 'Best First'.
Re^10: memory leaks with threads by misc (Friar) on Jul 10, 2007 at 13:34 UTC
I submitted a bugreport. I did some benchmarking, and calculated the time of running a function 300 times. The results are, again, most suprising to me. With 10 worker threads in a pool I get this: `micha@laptop ~/prog/perl/test $ time ./threads_benchmark.pl real 0m36.666s user 1m11.872s sys 0m0.172s (resident memory 6.3MB)` [download] Read more... (12 kB) And, creating and destroying 300 threads, 10concurrent threads `micha@laptop ~/prog/perl/test $ time ./threadpool.pl real 0m33.001s user 1m4.728s sys 0m0.024s (max res memory indefinite rising, 10MB)` [download] Read more... (1117 Bytes) Ok, my threadpool implementation is far from beeing perfect, but I had guessed the differences in speed would be more obvious. Btw, I benchmarked also Thread::Pool Thread::Pool `micha@laptop ~/prog/perl/test $ time ./thread_pool_benchmark.pl real 0m36.105s user 1m9.780s sys 0m0.048s 15M res memory` [download] Read more... (1186 Bytes) My thread pool implementation (the first benchmark and code listed) seems to be stable and not too bad, I believe there are already some advantages to Thread::Pool. Is there a proposed way to publish such code, perhaps in Snippets ? Update: Quoted the scripts in readmore tags, as suggested, I'm also going to publish my ongoing work in my scratchpad Comments welcome	[reply] [d/l] [select]
Re^11: memory leaks with threads by BrowserUk (Patriarch) on Jul 10, 2007 at 13:57 UTC
Is there a proposed way to publish such code, perhaps in Snippets ? Personally, I think adding your code (all 3 versions) to your post above in separate `<readmore><code>...</code></readmore>` blocks would be good. It'd put the benchmark figures into context, and make the code available for searching for those that follow. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply] [d/l]