(shared) memory and preloading modules using Mod-perl

jbrugger has asked for the wisdom of the Perl Monks concerning the following question:

Hi all. I have a reasonable large application, that runs under Mod-perl on Apache2.
Here I try to preload the modules, so they are loaded into the primary process of Apache. However, after a few requests, all the precompiled modules use about 60 megabytes of memory (15 mb per process), and it seems that each apache-child is having it's own set compiled.
Is there a way to let mod-perl share it's compiled modules from the main proces?

Next i tried to have only the most-usedmodules loaded, and cached by mod-perl, and the less used (and bigger) modules compiled run-time.

I tried to remove the namespace by
undef(%INC{modulename}),
used 'no modulename' and
do ('module.pm'), however the apache::status still tells me that the modules remain loaded.

Is there a way to let mod-perl load a module and free it's memory after use? I know there is the drawback of recompiling it each time we access it, but it is for some bigger and less used modules that i don't want to keep in memory. (ps: i don't want to run them in a non-modperl environment) The preferred solution would be to have the compiled modules shared (so the total application would take about 15 mb at the first startup of apache), but keeps the rest for the server (and application itself). Thanks in advance, Justus Brugman

Comment on (shared) memory and preloading modules using Mod-perl

Replies are listed 'Best First'.
Re: (shared) memory and preloading modules using Mod-perl by perrin (Chancellor) on Dec 29, 2004 at 18:32 UTC
Are you running with a threaded MPM? That uses a lot more memory. If you run with prefork MPM, the modules you load before forking will be shared by copy-on-write. This is all documented here.	[reply]
Re^2: (shared) memory and preloading modules using Mod-perl by jbrugger (Parson) on Dec 29, 2004 at 18:46 UTC
Thanks. I did not know that, but i'm using the prefork MPM on one machine and threaded MPM on the other. I thought it would make no difference to mod_perl, but i'll try it out.	[reply]
Re^3: (shared) memory and preloading modules using Mod-perl by perrin (Chancellor) on Dec 29, 2004 at 19:13 UTC
Threading wastes a lot of memory because perl threads are unable to share their opcode tree as effectively as copy-on-write does. If you have an OS that supports prefork, you should stick with prefork.	[reply]
Re: (shared) memory and preloading modules using Mod-perl by nacka (Friar) on Dec 29, 2004 at 21:32 UTC
Keep in mind that big data structures your code builds up aren't shared, since they are build up in the child processes. If you have big static data structures you could build them in a file PerlRequired from apache, at least then they become shared. Remember to destroy/free all big dynamic data structures. There's a few tips on measuring memory usage in Measuring Code Memory Usage from the Practical mod_perl book, there's a chapter on mod_perl2 specifics too.	[reply]
Re: (shared) memory and preloading modules using Mod-perl by ides (Deacon) on Dec 29, 2004 at 18:57 UTC
While this might kill your performance in other ways, you could choose to not preload the larger modules and then instruct the Apache child to exit. This would increase your forking of Apache children, but keep your memory use under control. You'd want to make sure and use CORE::exit() and not just exit() as mod_perl redefines the function to do exactly the opposite of what you want in this case. Frank Wiles <frank@wiles.org> http://www.wiles.org	[reply]
Re: (shared) memory and preloading modules using Mod-perl by gmpassos (Priest) on Dec 30, 2004 at 01:53 UTC
Take a look at Safe::World, where you can run your code inside a compartment than clean it from the memory. Also you can have many compartments as you want at the same time, each with different packages loades, or even assemble this compartments. You also can try to make a different architecture in your application, where you can let some part of your application as a service in an unique process, so, the other process will access this service without load it, just making a call using any RPC protocol. So, take a look at SOAP. About the prefork resource, is very nice if your OS has fork, but note that is not any module/code that can survive from a fork. Graciliano M. P. "Creativity is the expression of liberty".	[reply]
Re^2: (shared) memory and preloading modules using Mod-perl by perrin (Chancellor) on Dec 30, 2004 at 05:37 UTC
You also can try to make a different architecture in your application, where you can let some part of your application as a service in an unique process, so, the other process will access this service without load it, just making a call using any RPC protocol. So, take a look at SOAP. You'd end up needing to handle concurrent requests there as well, and probably writing your own forking daemon of some kind. Not worth it, unless you can use a single-process model. About the prefork resource, is very nice if your OS has fork, but note that is not any module/code that can survive from a fork. The same is true of threads, i.e. don't fork or spawn a thread with an open socket or some XS data structure and then try to use it from the new process/thread.	[reply]
Re: (shared) memory and preloading modules using Mod-perl by jbrugger (Parson) on Dec 30, 2004 at 07:49 UTC
He, it also is a versioning thing... When using the latest version of mod-perl, you can use: # removes a stash (.so, %INC{$stash}, etc.) as best as it can ModPerl::Util::unload_package($stash); This is not available in <= versions 1.99 (it's included in mod_perl-2.0.0-RC2-XMas), so mind the version of mod-perl on your servers when using it.	[reply]
Re: (shared) memory and preloading modules using Mod-perl by jbrugger (Parson) on Dec 30, 2004 at 06:18 UTC
First of all, Thanks for all the input you've given. For me, i will end up using the prefork MPM, and preload most of the modules. Concerning the memory use of the application, i think Safe::World might help me, i'll look into that. But for an answer to the question: Can i (force) unload a compiled module from mod-perl, and give back it's memory to the os, the answer would be no i'd guess. ps. "You'd end up needing to handle concurrent requests there as well, ..." Would it not just be an idea to do a request using a unique ticket (id), so all the request are done using some sort of hash? to ensure that the processes could be asynchrone, i'd do a request(uniqueMD5_ID) and expect the result in waitForTheAnswer(uniqueMD5_ID) Thanks agaim. and have a nice new year all.	[reply]