MrOreo has asked for the wisdom of the Perl Monks concerning the following question:

Hey perl monks around the world!
For the program that I'm writting I have to use multi-dimensional hash tables, and I have to save them to a file at the end of the program, and then include the save file at the starting of the next program.
The way I save the hash tables, is I basically create a huge perl file with lots of lines that say
$sites{"www.l33t.ca"}{'files'}{'/cgi-bin/whatever.cgi'}{'hits'} = 2034
+;

[download]

and the like. At the start of the program when I 'require' the file, it runs fine, but takes up a whooping 180 megs of RAM to do so!! The save file is only 14 megs large, and when I generate the file, the perl interpreter only uses at maximum, 20 megs of ram..
How can I 'require' this large file with out taking up so much damn memory!

Thanks a lot!

-MrOreo
www.l33t.ca

Replies are listed 'Best First'.
Re: Memory Optimization
by IlyaM (Parson) on Dec 30, 2001 at 03:17 UTC
    Don't use huge perl files. If you need to store multi-level huge hash structure then use MLDBM.

    --
    Ilya Martynov (http://martynov.org/)

[reply]
Re: Memory Optimization
by Zaxo (Archbishop) on Dec 30, 2001 at 03:35 UTC

    IlyaM is right, but if you must follow this design, try Data::Dumper. Make each file contain a single object, so that $foo = do "foo.pl"; returns a reference to the struct. That will minimize copying. It will probably be useful to read them in a BEGIN block.

    After Compline,
    Zaxo

[reply]
[d/l]
Re: Memory Optimization
by jlongino (Parson) on Dec 30, 2001 at 05:57 UTC
    For a complex hash you'll probably want to use Storable qw(nstore);. nstore is more portable than store and you can still use the same retrieve function. You might need to experiment but below are snippets from code that I currently use. The resulting file is roughly 2.2 times larger than a fixed length text file of the same data. Excerpts from The Perl Cookbook gives some explanation. To save the hash structure to disk: 
    use strict;
use Storable qw(nstore);

my %hash = my_build_hash_sub();
nstore (\%hash, "hash.nstore.dat");

    [download]
    To reload hash from disk: 
    use strict;
use Storable;

my %hash = %{ retrieve("hash.nstore.dat") };

    [download]

    --Jim

[reply]
[d/l]
[select]
      Speaking of memory optimizations, my %hash = my_build_hash_sub(); will toss around a list and eat memory for that. Returning a reference to a hash would in many cases be more efficient than returning a list of keys/values. 

      2;0 juerd@ouranos:~$ perl -e'undef christmas'
Segmentation fault
2;139 juerd@ouranos:~$

      [download]

[reply]
[d/l]
        You're absolutely right. My concern when these snippets were written was not memory optimization so much as decreasing IO read/writes times.

        Thanks for the clarification. I'll look into rewriting the hash generation subs to use references.

        --Jim

[reply]

Back to Seekers of Perl Wisdom