Big objects: Dumper vs. Storable

nop has asked for the wisdom of the Perl Monks concerning the following question:

It seems that for large objects, Storable can work where Dumper runs out of memory: in the following code, dumpsave works, and dumpsave2 fails (applied on the same monster object, a nested hash structure with around 100K leaves).

I like Dumper better, because you can read the resulting file.

Anyway to coax Dumper to handle big objects?

sub dumpsave {
    my ($fname, $ref)= @_;
    print "saving structure to $fname...\n";
    store $ref, $fname;
}

sub dumpsave2 {
    my ($fname, $ref)= @_;
    print "saving structure to $fname...\n";
    local $Data::Dumper::Indent = 0; 
    local $Data::Dumper::Purity = 1; 
    local $Data::Dumper::Terse = 1;  
    local $Data::Dumper::Deepcopy = 1;
    open (F, ">$fname") or die "cant write open $fname";
    print F Dumper($ref);
}
[download]

Comment on Big objects: Dumper vs. Storable Download Code

Replies are listed 'Best First'.
(tye)Re: Big objects: Dumper vs. Storable by tye (Sage) on Oct 16, 2000 at 22:19 UTC
My biggest gripe with Data::Dumper is that is keeps appending to a big string in memory when 90% of the time you are just going to write that string to a file. Someone needs to patch Data::Dumper so that you can request that it dump directly to a file handle (that and it needs to be more flexible in what it puts on the left-hand side of the equals sign). Oh, and it should have an option to not dump the blessedness. - tye (but my friends call me "Tye")	[reply]
(isotope) Re: Big objects: Dumper vs. Storable by isotope (Deacon) on Oct 25, 2000 at 02:59 UTC
I agree with tye completely on this one, especially since I'm trying to dump a very large hash to disk simply to examine its contents by hand. My Sparc Ultra 10 didn't like the request for 268MB RAM. --isotope http://www.skylab.org/~isotope/	[reply]