| [reply]
[d/l] |
Its a pretty common problem on linux, with Perl threads. The best solution I have come up with, is to reuse the threads. You set a max number of threads to be handled at any one time, then feed them data to process thru threads:shared variables, and when they are done, don't try to destroy them; instead, clean out their old data, and reuse them, as if they are in a circular worker queue. See Reusable threads demoThat way, your memory will increase for say 20 worker threads, then will stay relatively constant. It seems reasonable programming style to reuse massive thread objects, rather than destroying and recreating them. Remember, filehandles can be shared across threads, thru the fileno.
| [reply] |
A classical solution of long running processes which consume more memory over time is do to an exec of itself very $N hours or $M days. Of course, depending on the application, this may not always be feasible. | [reply] |
As Zentara suggested, there must not be a one-to-one correspondence between threads and requests.
Consider a fast-food restaurant. Every time someone places a new order, a new employee does not materialize out of thin air to cook the food, prepare it, and then fall over dead. No, the entire system is designed around a carefully arranged system of queues, and a limited number of workers, most of whom can handle many different chores.
You need to arrange to have a limited (and controllable) number of threads, all of whom continue to exist even if they have no work to do. If the system has more work to do than the current number of threads can handle, then some of that work must wait in queues ... as a mere description of work that is to be done in the near future.
| |
Perl Threads are a joke. The perl engine NEVER releases RAM to the OS. Why? Who knows. One explanation I've heard is RAM fragmentation or that perl never sees a totally free block of RAM again to hand back to the OS. The other is that perl does its own memory management, and it mallocs huge chunks of ram from the OS, and will only "free" them back to its internal memory allocator. The internal memory allocater will NEVER free them back to the OS. Its the problem of perl being the only implementation of Perl.
$t = "perlperlperlperlperlperlperlperlperlperlperlperlPP" x 20000; #1
+ MB
for (0..100) {$t3 .= $t;}
system('pause');
undef($t3);
undef($t);
system('pause')
The code above shows that perl does release ram back to the OS according to task manager. It went to 200 MB of RAM in task manager on the 1st stop, after the 2nd stop it was at 13MB RAM. So in single threaded mode, perl will release RAM back to the OS on Windows.
Ithreads are a joke. They receive little attention from perl developers since no "real" Perl user cares about threads since "all" "real" Perl users use Perl for CGI under POSIX which includes COW fork support and daemonization which remove the need for threads support in perl in perl developers minds/priorities. Perl ithreads is a 1 or 2 man job according to the readme, just 2 developers listed. Another thing that shows that perl developers only care about use perl for CGI and not for the desktop is that apache modperl compiles Perl to bytecode and saves the bytecode for later execution. The desktop version of this were B::C and Byteloader modules, which were removed in 2007 from the CORE since they weren't maintained and development stopped on them for a long time. Perl on the desktop and especially Windows receives very little support overall. From what I've seen, Activestate used to spend alot of money on perl on win32 in the late 1990s/early 2000s, but has given up on investing in perl since then. Activestate used to contribute a huge amount to the Perl world. Win32 patches, PerlNET, PerlScript, Perl in IIS, and countless other things.
Regarding your threads memory consumption problem, [rt://44493] talks about it. So does the POD for threads. Note the author of ithreads basically said he doesn't care about the problem/mem leak and its your fault and he isn't going to fix it. Fits the attitude of poor/no support of Perl on the desktop.
The problem with ithreads is that ithreads doesn't share 1 perl interpretor/op tree executor between multiple threads. Each perl ithread gets its own perl engine copy of x86 assembly in RAM and basically is a new perl process embedded in a new thread created in XS. A better design, in my amateur opinion would be 1 static perl interpreter, with each thread having its own stack/context and optree, but that won't happen since perl developers are perfectly happy with COW fork on POSIX and won't change perl to work on Windows.
Cygwin fork doesn't COW either. I'm not sure about Interix fork. Interix is the damn closest you can get to POSIX on windows. Interix is closed source, and its developers have access to the Windows source code and it uses the Native NT API that Cygwin and not other FOSS POSIX toolkits get to use. Try using perl on Interix and then doing fork and see how your RAM usage is compared to ithreads. Under the hood NT series is Unix or has all provisions to be Unix remember. The NT Kernel was made by a bunch of layed off DEC Unix engineers. The NT Kernel API is undocumented and unsupported by Microsoft, on purpose probably. Win32 API isn't Unix obviously.
In a normal programing language, there would be multiple Perl interpreters, and a Perl standard (although a fixed standard just wouldn't ever work with Perl ;)). Look at Javascript. Multiple engines. Multiple toolkits. Multiple enviroments. Look at Javascript, HTML, all Adobe products, Openoffice, JScript, J# (RIP), QTScript, all use Javascript, but are totally different things. Its sad that Perl is 23 years old, yet there is no IronPerl for .NET or any way to compile to x86 or bytecode on the desktop or anything but perl to run Perl. Perl 6 is a big part of the problem. Aslong as there is Perl 6, Perl and perl are a dead end. Its sad in 23 years Perl has had 2 forks (Ruby and PHP) because of developers being fed up with it for various reasons (someone elaborate if they have a plausible explanation).
perl also has no asynchronous File I/O on windows, well, it does, but its basically is undocumented in Win32API::File and has some cosmetic bugs. I've gotten it to work perfectly with workarounds for the cosmetic bugs. Putting my notes for doing async file I/O on Win32 on the internet is on my TODO list.
Your fix as said in ithread's POD is to reuse threads. Or just launch multiple perl.exes. What about using select() to handle multiple connections with a non-threaded perl? It supposedly work on win32.
In an ideal world, threads are NEVER needed in any algorithm, unless you want to use multiple processors. Before using the permabroken ithreads, ask yourself do you really need ithreads in the first place?
I heard there is an opinion that ithreads can never be used in a "production environment", and they exist solely for testing or developing POSIX Perl scripts on Windows, later to be transferred to a POSIX server for the "production environment".
Also what other modules are you using other than ithreads in your server Perl script? If LWP is one of them, thats your problem. | [reply]
[d/l] |
I found in my application that we have created threads and which not joined anywhere.
Now I have created another thread which will join the threads. After this one I got some what better memory usage. The aggressive memory eating by my application getting reduced now.
sub MemoryCleanup
{
while(1)
{
foreach(threads->list())
{
$_->join() if($_->is_joinable());
}
sleep(5);
}
}
| [reply]
[d/l] |
I found in my application that we have created threads and which not joined anywhere.
Yes. That would cause runaway memory usage.
Whilst your solution above may work for you, there are much better ways to deal with the situation.
One possibility it to detach the threads when you create them:
threads->create( \&yourSub )->detach;
...
Now the threads will go away automatically, as soon as they end and your extra thread above will be unnecessary.
If you'd answer a few questions here, there are other, better possibilities also.
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply]
[d/l] |
| [reply] |