Re^2: problems with fork from thread

Forking is not an ideal solution, because I exchange data between the threads and I will have to rewrite a lot to go with forks.

If I use only threads, I have no problems.

My problem is that I'm using n threads doing work with HTML::TreeBuilder. Somewhere in there I have some memory leaks, because the memory used by the script is increasing until it will be killed because of OOM.

So my solution (that will not involve rewriting the entire application with forks) was to fork only the HTML::TreeBuilder part and so I will get read of the accumulated memory leaks.

Seems I hit an other problem and I have to look on a way to go with fork and sockets.

I still hope that a solution will appear for this.

Comment on Re^2: problems with fork from thread

Replies are listed 'Best First'.
Re^3: problems with fork from thread by BrowserUk (Patriarch) on Mar 02, 2012 at 07:06 UTC
May I ask if you are successfully using threading for your program, why to you want to fork as well? What are you going to do after you fork? Note: I'm not condemning the use of fork here, just trying to see a way to a solution for you that avoids needing to re-write your entire application. If, for example, you need to run an external command, then there are a couple of ways of approaching that that are known to work. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply]
Re^4: problems with fork from thread by cristi1979 (Novice) on Mar 02, 2012 at 07:40 UTC
Like I said, I'm forking in order to not accumulate the memory leaks in the main program. Forking will increase the memory size of the new process only and get rid of it when it dies. I have $tree = $tree->delete() and also undef ($tree). Also I put everything in a class and undefed the class, but I still see the memory increasing.	[reply]
Re^5: problems with fork from thread by Gangabass (Vicar) on Mar 04, 2012 at 02:49 UTC
Also I have moved `use HTML::TreeBuilder::XPath;` to the thread (with `require`) and this help me...	[reply] [d/l] [select]
Re^6: problems with fork from thread by cristi1979 (Novice) on Mar 06, 2012 at 08:22 UTC
Re^3: problems with fork from thread by Gangabass (Vicar) on Mar 02, 2012 at 06:42 UTC
I have same problem recently (threaded app with `HTML::TreeBuilder::XPath`). I have downloaded and processed less than 6000 entries (html pages) but my program uses 2Gb! I have fixed this issue by adding `$tree->delete()` after I don't use it. Thanks. Roman	[reply] [d/l] [select]