Re: Large Storable files and Memory Woes

What's your access pattern for the data? If you don't need it all in memory at once, a lightweight database like DBM::Deep or even a dbm file might work better. If you do need it all in memory at once, you need more memory.

Comment on Re: Large Storable files and Memory Woes

Replies are listed 'Best First'.
Re^2: Large Storable files and Memory Woes by onelesd (Pilgrim) on Jun 05, 2011 at 22:56 UTC
Thank you for your response. Unfortunately, the access pattern is irrelevant at this point, as I only have the data in Storable format, and now need to convert it to another format (without the read-all-at-once problem) in order to read it. Storable was the wrong format to store the data in to begin with, but now I am stuck with it as my source. I have been unsuccessfully trying `grep -a` so far.	[reply]
Re^3: Large Storable files and Memory Woes by GrandFather (Saint) on Jun 06, 2011 at 01:13 UTC
Take a look at the source for the specific version of Storable that you used and extract the code you need to recover your data. It sounds like you have stored the contents of a simple hash containing simple scalar keys and values (numbers and strings) so the amount of code you need to emulate or lift from the Storable source should be fairly small. Once you've done that of course you should consider a different technique for persisting the data. True laziness is hard work	[reply]