Large Storable files and Memory Woes

onelesd has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Large Storable files and Memory Woes by chromatic (Archbishop) on Jun 05, 2011 at 22:44 UTC
What's your access pattern for the data? If you don't need it all in memory at once, a lightweight database like DBM::Deep or even a dbm file might work better. If you do need it all in memory at once, you need more memory.	[reply]
Re^2: Large Storable files and Memory Woes by onelesd (Pilgrim) on Jun 05, 2011 at 22:56 UTC
Thank you for your response. Unfortunately, the access pattern is irrelevant at this point, as I only have the data in Storable format, and now need to convert it to another format (without the read-all-at-once problem) in order to read it. Storable was the wrong format to store the data in to begin with, but now I am stuck with it as my source. I have been unsuccessfully trying `grep -a` so far.	[reply]
Re^3: Large Storable files and Memory Woes by GrandFather (Saint) on Jun 06, 2011 at 01:13 UTC
Take a look at the source for the specific version of Storable that you used and extract the code you need to recover your data. It sounds like you have stored the contents of a simple hash containing simple scalar keys and values (numbers and strings) so the amount of code you need to emulate or lift from the Storable source should be fairly small. Once you've done that of course you should consider a different technique for persisting the data. True laziness is hard work	[reply]
Re: Large Storable files and Memory Woes by BrowserUk (Patriarch) on Jun 06, 2011 at 02:53 UTC
How valuable is this data to you? How private is this data? How big is(are) the file(s)? Do you remember exactly (or roughly) how many keys there are? Which versions of perl & Storable was the data stored under? Was it `store`d or `nstore`d ? If the former, what hardware was it stored on? (32-bit intel; 64-bit Solaris; other?) What do typical keys and values look like? Eg.`fred => 123` or `fred => 123.456` or `fred => 'bill'` or `123 => 'fred'`? Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l] [select]
Re: Large Storable files and Memory Woes by Khen1950fx (Canon) on Jun 06, 2011 at 08:22 UTC
On my Linux system, when I encounter too much swapping, I can reduce that by resetting "swappiness". For example, to see what the system default is: `sysctl vm.swappiness` [download] On my system the default is 60. I have 512mb ram. So if i want to retrieve a file that is 4G, then I would reset swappiness to 0 to speed things up. To reset, just: `sysctl -w vm.swappiness=0` [download] After the retrieval is finished, reset back to default.	[reply] [d/l] [select]
Re: Large Storable files and Memory Woes by locked_user sundialsvc4 (Abbot) on Jun 06, 2011 at 10:22 UTC
I have never been too happy with Storable, in the best of times. I have too many times been “bit” by things that I froze that I could not subsequently thaw. If you have large files to deal with, I suggest that you use a format like XML, and that you use a full-blown package like XML::Twig which is capable of extracting data from even a “very, very large” file, i.e. without having to read it all into RAM. That might not help you now, if you no longer have access to the machine, but it is a good long-term approach, methinks.
Re^2: Large Storable files and Memory Woes by onelesd (Pilgrim) on Jun 09, 2011 at 06:44 UTC
Aye, I have learned my lesson regarding Storable as a long-term storage format. In the future I will use JSON or XML, since these can be easily parsed without needing to read the whole file into memory. I was able to finally extract the data through some creative usage of grep and what I knew about the stored data. Once again regular expressions save the day.	[reply]