Safe.pm and memory problems

shushu has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks !
I am using the modules Safe and Safe::Hole in order to execute code I am getting from external resources.
The same Perl code is going in a loop over a list of files, execute each of them, and this way check it plus minus.
As long as the code being executed is straight Perl script there is no problem, but this is the behavior when the script uses external modules -

In case I use a different name space for every script executed - then every module used is loaded to the memory, and even after I finish the use of the module and undef the Safe object being used - it seems as though the module is not removed from the memory. This cause kind of memory "leak", since when executing a long list of modules the Perl process memory consumption is getting bigger and bigger.
In case I use the same name space for all of the scripts executed - then every module use is loaded to the memory once as well, and even undefing the Safe compartement make no different. The main problem here is that in some cases the script needs the module to be loaded again (due to BEGIN code, for example), and that if several scripts uses different modules but by the same package name the script will actually will use a wrong package (the @INC for every script is created in the execution time and defined temporarily to create higher separation. This is why it might be the same package name with a different file).
In case I try harder, use the same name space but undef the name space %INC variable between every script - it solves the loading of modules so the modules used are being loaded to the memory every time, bt it cause a memory problem as well - it seems that although the %INC is cleaned Perl still keeps the old loaded module copy in the memory and does not clean it until the end of the whole process.

The bottom line is -
I need to find a way to clean every piece of memory used by a former executed script in the Safe compartement in order to execute the script after it. It looks as though I will need to clean the name space itself and not the Safe object, and any new name spaces created while the script is being executed.
Since Perl's garbage collector, as far as I know, is being executed automatically without any way to force it, it seems I need to find a way to know what "data" Perl has in his memory a in order to "undef" it one by one.

Questions are -

Is therer any way to force Perl to "garbage collect" something ?
Is there a way to know what variables exists in a specific name space ?
Is there a way to find what modules were loaded, and remove them from the memmory ? (cleanning the %INC is not enough)

I don't see any specific reason to send any code, but if it is necessary I will gladly do it.

Thanks all,
shushu

Comment on Safe.pm and memory problems

Replies are listed 'Best First'.
Re: Safe.pm and memory problems by integral (Hermit) on Feb 09, 2003 at 08:38 UTC
`%INC` only contains a mapping from a module name like 'strict.pm' to its full path (Eg. '/usr/share/perl/5.6.1/strict.pm'). Perl uses it in requires and uses to ensure that you can only load a module once. Therefore clearing %INC will only reset perl's idea of what has been loaded and not actually remove any module code from memory. Removing a module from memory can be as simple as undefing its symbol table like (UPDATE: but only if this actually worked) `undef %strict::;` [download] Although when using Safe you need to do various things to find the right root namespace and delete the module inside of it. You can find out more about the way that perl exposes its package symbol tables to the outside world in perlmod. After reading this you should understand the code below that lists all the variables in a given namespace. `sub symbols { my $pkg = $_[0] \|\| ''; no strict 'refs'; print "Symbols: ", join(" ", sort keys %{$pkg.'::'}), "\n"; } symbols('strict');` [download] This routine and others are featured in the 'Displaying symbol tables for debugging' snippit. -- integral, resident of freenode's #perl	[reply] [d/l] [select]
Re: Re: Safe.pm and memory problems by IlyaM (Parson) on Feb 09, 2003 at 21:00 UTC
Removing a module from memory can be as simple as undefing its symbol table like `undef %strict::;` Unfortunately it doesn't work. See undef Package;. -- Ilya Martynov, ilya@iponweb.net CTO IPonWEB (UK) Ltd Quality Perl Programming and Unix Support UK managed @ offshore prices - http://www.iponweb.net Personal website - http://martynov.org	[reply] [d/l]
Safe.pm and memory problems - continue... by shushu (Scribe) on Feb 16, 2003 at 12:45 UTC
Hi Monks! I am coming back with questions about Safe.pm and memory handling, after reviewing the following messages: 'Displaying symbol tables for debugging' - and getting the symbol2 and packages subroutines from it. 'undef Package;' (156281) - and using Symbol delete_package subroutine. So I created a small script checking using the information gathered. The script uses Symbol.pm and Safe.pm, and include symbol2 and packages subroutines. All it does is execute a Perl script that uses a module (big module), and check the memory consumption. The script: use Safe; use Symbol qw(delete_package); use Data::Dumper; use strict; my $rc = 0; my $namespace = "SPACE"; my $CPT = new Safe($namespace); $CPT->deny_only(); my $file = shift; print "DEBUG - before rdo-\n"; system ("ps auxww\|grep safe\|grep -v grep"); print "-"x40,"\n"; <>; if (-f $file) { $CPT->rdo($file); } else { print "File \"$file\" does not exists !\n"; $rc = 1; } print "DEBUG - after rdo-\n"; print $@ if (defined $@); print "Printing \%SPACE::\n"; print Dumper \%SPACE::; print "Printing \%SPACE::INC\n"; print Dumper \%SPACE::INC; print "\nOutput from space_packages\n"; space_packages(); print "\nOutput from packages\n"; packages(); print "\nOutput from symbols2\n"; symbols2("SPACE"); system ("ps auxww\|grep safe\|grep -v grep"); print "-"x40,"\n"; <>; delete_package($namespace); print "DEBUG - after clearing namespace-\n"; print "Printing \%SPACE::\n"; print Dumper \%SPACE::; print "Printing \%SPACE::INC\n"; print Dumper \%SPACE::INC; print "\nOutput from space_packages\n"; space_packages(); print "\nOutput from packages\n"; packages(); print "\nOutput from symbols2\n"; symbols2("SPACE"); system ("ps auxww\|grep safe\|grep -v grep"); print "-"x40,"\n"; <>; sub space_packages { print "Packages: ", join(" ", grep /::$/, keys %SPACE::), "\n"; print " INC: ", join(" ", sort keys %SPACE::INC), "\n"; } sub packages { print "Packages: ", join(" ", grep /::$/, keys %::), "\n"; print " INC: ", join(" ", sort keys %INC), "\n"; } sub symbols2 { my $pkg = $_[0] \|\| ''; no strict 'refs'; print "Symbols in ${pkg}::\n"; my $hash = {$pkg.'::'}{HASH}; for (sort grep {!/::$\|^_</} keys %$hash) { my $sym = $_; my $glob = $hash->{$_}; $sym =~ s/[[:cntrl:]]/'^' . chr(ord($&) + ord('@'))/ge; print " $sym (", {$glob}{PACKAGE}, '::', {$glob}{NAME}, "): "; if (defined $$_) { print "SCALAR='$$glob' "; }; for my $THING (qw(ARRAY HASH CODE IO)) { my $ref = {$glob}{$THING}; if (defined $ref) { print "$THING=$ref "; } } print "\n"; } } [download] The output (executed on Linux RH 7.3 with Perl 5.6.1): [root@gili /tmp]# perl safe.pl /tmp/safe2.pl DEBUG - before rdo- root 31983 0.0 1.6 3504 2068 pts/0 S 06:26 0:00 perl sa +fe.pl /tmp/safe2.pl ---------------------------------------- DEBUG - loading module momo- test DEBUG - after rdo- Printing %SPACE:: $VAR1 = { 'INC' => SPACE::INC, '_' => ::_, 'a' => SPACE::a, '_</tmp/safe2.pl' => {'SPACE::_</tmp/safe2.pl'}, '__ANON__' => SPACE::__ANON__, 'test' => SPACE::test, 'momo::' => {'SPACE::momo::'}, '_<momo.pm' => {'SPACE::_<momo.pm'}, 'BEGIN' => SPACE::BEGIN, 'main::' => {'SPACE::main::'} }; Printing %SPACE::INC $VAR1 = { 'momo.pm' => 'momo.pm', '/tmp/safe2.pl' => '/tmp/safe2.pl' }; Output from space_packages Packages: momo:: main:: INC: /tmp/safe2.pl momo.pm Output from packages Packages: SPACE:: Opcode:: DB:: Data:: DynaLoader:: Exporter:: warning +s:: Symbol:: strict:: XSLoader:: attributes:: UNIVERSAL:: overload:: +<none>:: Safe:: subs:: IO:: Carp:: CORE:: main:: INC: Carp.pm Data/Dumper.pm Exporter.pm Exporter/Heavy.pm Opcode.pm +Safe.pm Symbol.pm XSLoader.pm overload.pm strict.pm subs.pm warnings. +pm warnings/register.pm Output from symbols2 Symbols in SPACE:: BEGIN (SPACE::BEGIN): INC (SPACE::INC): HASH=HASH(0x81945d8) _ (SPACE::_): SCALAR='_' ARRAY=ARRAY(0x819e1c8) __ANON__ (SPACE::__ANON__): a (SPACE::a): test (SPACE::test): CODE=CODE(0x819b320) root 31983 13.0 2.4 4500 3068 pts/0 S 06:26 0:00 perl sa +fe.pl /tmp/safe2.pl ---------------------------------------- DEBUG - after clearing namespace- Printing %SPACE:: $VAR1 = {}; Printing %SPACE::INC $VAR1 = {}; Output from space_packages Packages: INC: Output from packages Packages: Opcode:: DB:: Data:: DynaLoader:: Exporter:: warnings:: Symb +ol:: strict:: XSLoader:: attributes:: UNIVERSAL:: overload:: <none>:: + Safe:: subs:: IO:: Carp:: CORE:: main:: INC: Carp.pm Data/Dumper.pm Exporter.pm Exporter/Heavy.pm Opcode.pm +Safe.pm Symbol.pm XSLoader.pm overload.pm strict.pm subs.pm warnings. +pm warnings/register.pm Output from symbols2 Symbols in SPACE:: root 31983 0.8 2.4 4500 3068 pts/0 S 06:26 0:00 perl sa +fe.pl /tmp/safe2.pl ---------------------------------------- [download] What can we see here ? The delete_package seems to work. BUT - the memory consumption does not getting low. Any suggestions, except "fork" ? Thanks, shushu	[reply] [d/l] [select]
Re: Safe.pm and memory problems - continue... by shushu (Scribe) on Feb 17, 2003 at 10:21 UTC
Re: Safe.pm and memory problems - continue... by diotalevi (Canon) on Feb 18, 2003 at 00:08 UTC
Re: Safe.pm and memory problems by diotalevi (Canon) on Feb 09, 2003 at 16:35 UTC
Take the easy way out: fork for each tested program. Turn your Safe+Safe::Hole script into a wrapper and just test a program once - return the value via IPC (probably just do a piped fork) and exit. Now you don't have to even worry about this problem. Seeking Green geeks in Minnesota	[reply]