in reply to Storable::freeze performance problem

I've run into the same problem (I believe), here's how I discovered it
=head1 Storable 2.15 has a memory leak Here's how I discovered it. First you go to where cpanplus keeps its configuration info (the .cpanplus directory) $cpanplusconf->{conf}{base} and delete the sourcefiles.*.stored file C:\Perl\.cpanplus>perl -e"die time" 1117533963 at -e line 1. C:\Perl\.cpanplus>cpanp CPANPLUS::Shell::Default -- CPAN exploration and modules installat +ion (v0.053) *** Please report bugs to <cpanplus-bugs@lists.sourceforge.net>. *** Using CPANPLUS::Backend v0.053. ReadLine support disabled. CPAN Terminal> l CGI Details for 'CGI' Author Lincoln D. Stein (lstein@cshl.org) Description None given Development Stage Unknown Interface Style Unknown Language Used Unknown Package CGI.pm-3.10.tar.gz Public License Unknown Support Level Unknown Version Installed 3.10 Version on CPAN 3.10 Contains: CGI CGI::Carp CGI::Cookie CGI::Fast CGI::Pretty CGI::Push CGI::Util CPAN Terminal> q Exiting CPANPLUS shell C:\Perl\.cpanplus> C:\Perl\.cpanplus>perl -e"die time" 1117533981 at -e line 1. C:\Perl\.cpanplus>perl -e"die 1117533981-1117533963" 18 at -e line 1. C:\Perl\.cpanplus>dir *stored 05/31/2005 03:06 AM 14,631,498 sourcefiles.2.15.stored This is cool (18 seconds), but if you repeat these steps, cpanp will +load sourcefiles.2.15.stored, but quitting will take a very very long time, eating %100 of the CPU, and increasingly more and more memory. I quit after the first time. I've traced through CPANPLUS, and it is the call to Storable::nstore that is to blame. =cut #use Pod::Usage;pod2usage(-verbose => 2); print "you should chdir where CPANPLUS keeps sourcefiles*.stored$/"; print "which is something like \$cpanplusconf->{conf}{base}$/"; print "and delete the sourcefiles.*.stored file$/"; print "This should take a short time (under a minute)$/"; my $now = time; system qw[ perl -S cpanp l CGI]; print time-$now,$/,$/; print $_,$/ for glob 'sourcefiles*.stored'; print "This should take a long time (many minutes)$/and leak memory$/" +; $now = time; system qw[ perl -S cpanp l CGI]; print time-$now,$/,$/;
I haven't been able to workout why it happens, but it's got something to do with my compiler setup and/or my perl version (5.8.4 and less).

It only happens with Storable 2.15 and up, so downgrading to Storable 2.14 seems to fix it, as does using (Storable-2.15+ with) perl 5.8.5 and up.

Corion says maybe it is a memory leak/allocation problem that was fixed in 5.8.5, he might be right.

MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
** The third rule of perl club is a statement of fact: pod is sexy.

Replies are listed 'Best First'.
Re^2: Storable::freeze performance problem
by waswas-fng (Curate) on Jun 24, 2005 at 17:49 UTC
    Thanks Pod, I ran into this same issue not to long ago and had to revert to the older version to fix it. I never had time to go back and investigate the issue, my guess was that the 2.15 or later versions had changes the recursive loop in some way that just did not jive with my app's data structure. ++ * 10 if I could.


    -Waswas
Re^2: Storable::freeze performance problem
by Smoothhound (Monk) on Jun 24, 2005 at 23:13 UTC

    Bless you PodMaster! reverting to 2.14 did the trick, it will be sometime before we move to 5.8.5+ here.

    Thawing, manipulating, presenting the report then freezing again for 3 data items for 20000 applicants now takes ~3 seconds instead of ~19!

    Thanks again, I've been banging my head against this one for a couple of days