Expected speed of retrieve function (nstore retrieve)

rjbioinf has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Expected speed of retrieve function (nstore retrieve) by BrowserUk (Patriarch) on May 16, 2012 at 11:30 UTC
When I load it back into Perl using 'retrieve' it takes > 5 minutes. That is very slow. Using a a 32-bit perl, nstoring a 40e6 (numeric) element array took 12 seconds and produces a 200MB file: `C:\test>\perl32\bin\perl.exe \perl32\bin\p1.pl use Storable qw[ nstore retrieve ];; $#a = 40e6; $a[ $_ ] = $_ for 0..40e6-1;; $t=time; nstore( \@a, '/test/junk.sto' ); print time()-$t;; 12.3673732280731 Terminating on signal SIGINT(2) C:\test>dir junk.sto 16/05/2012 12:22 199,999,628 junk.sto` [download] Retrieving it took 12 seconds and used 1.1GB peak: `C:\test>\perl32\bin\perl.exe \perl32\bin\p1.pl use Storable qw[ retrieve ];; $t=time; $a = retrieve( '/test/junk.sto' ); print time()-$t;; 12.2560520172119` [download] A couple of possibilities: Is the disk where you are storing it local? Heavily used? On its last legs? Is loading the data (briefly requiring 1.1GB) moving you into swapping? With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply] [d/l] [select]
Re: Expected speed of retrieve function (nstore retrieve) by Eliya (Vicar) on May 16, 2012 at 11:22 UTC
`perl -e'$x[$_]=42 for 1..40_000_000; <>'` [download] already needs around 1.5GB memory on my machine, and `perl -e'$x[$_]="foo" for 1..40_000_000; <>'` [download] takes 3.3GB (checked with `top`). So depending on what exactly you have in your array, and whether the process of retrieving involves making another copy of the data on the stack (not that unlikely, though I haven't checked), and what else you have running on your machine, you might well be running out of physical memory...	[reply] [d/l] [select]
Re: Expected speed of retrieve function (nstore retrieve) by Anonymous Monk on May 16, 2012 at 11:40 UTC
For anybody else wondering, `&nstore` & `&retrieve` come from Storable module.	[reply] [d/l] [select]
Re: Expected speed of retrieve function (nstore retrieve) by sauoq (Abbot) on May 16, 2012 at 12:02 UTC
The resulting dump file is 90Mb (40 million elements). What kind of elements? `-sauoq "My two cents aren't worth a dime.";`	[reply]
Re^2: Expected speed of retrieve function (nstore retrieve) by rjbioinf (Acolyte) on May 16, 2012 at 12:10 UTC
Just integer numbers. I think it is a memory issue. I shut down other memory greedy applications and the retrieve process runs faster at ~40 seconds. I think OSX Lion has problems with freeing up memory.	[reply]
Re^3: Expected speed of retrieve function (nstore retrieve) by kcott (Archbishop) on May 16, 2012 at 18:36 UTC
There's an application for Mac OS X called "FreeMemory" which, as the name suggests, frees up memory. I've been using it for 6 months or so without any problems. It was a free download from the AppStore. You may find it useful. -- Ken	[reply]