Saving a hash/array to a file

davehorsfall has asked for the wisdom of the Perl Monks concerning the following question:

Perl 5.16.3 and 5.10.0 on MacBook Snow Leopard.

Dear Masters of Puissance,

Is there a way to write an array or a hash to a file for later fast retrieval? I'm playing with dictionaries, and the slowest part of the operation is reading it in (yes, I've profiled it).

More specifically, I have a hash whose key is the sorted letters in a word (as an aside, sorting those letters with split/sort/join is about 1/3rd of the running time) and the value is, wait for it, a reference to an anonymous list of its anagrams.

It would be cool if the hash thus constructed could be dumped out to a file, for immediate reloading next time around. I understand that whatever technique is chosen would be heavily Perl-dependant, but it would be a simple matter to detect this, e.g. a header indicating the Perl used to create it, and rebuilding it.

Tied variables seem to suggest themselves, but I can't quite grok them (I've been a C programmer far longer than I've been a Perl programmer, and I really love Perl!).

Ta muchly.

-- Dave

Comment on Saving a hash/array to a file

Replies are listed 'Best First'.
Re: Saving a hash/array to a file by Anonymous Monk on Aug 22, 2014 at 07:48 UTC
DB_File or BerkeleyDB or Storable	[reply]
Re^2: Saving a hash/array to a file by dcronin135 (Acolyte) on Sep 24, 2014 at 01:44 UTC
Posted a similar solution, that may help. DocID:1101591	[reply]
Re: Saving a hash/array to a file by RichardK (Parson) on Aug 22, 2014 at 10:38 UTC
This being perl there are lots of ways you could do it. It's going to depend on exactly your data structure and personal preference. You could use Data::Dump, so :- `use Data::Dump qw(dump); $str = dump(@list); .... @copy_of_list = eval $str;` [download] But, I prefer to use JSON as the output is human readable, and I can easily edit it.	[reply] [d/l]
Re: Saving a hash/array to a file by flexvault (Monsignor) on Aug 22, 2014 at 10:05 UTC
davehorsfall, While AM mentioned using a DB, a very simple way to save a hash/array to a file, is to use a 'foreach' or 'for' loop on the hash or array and print the contents to a file. When you need the hash/array again just use the same technique in reverse. You don't mention the type of data, but if it's ASCII text, you can use the tab character (\t) between the key and value with the normal carriage return (\n) to end the value field. But since it's speed you require, I would use the unbuffered 'sysopen/sysread/syswrite'. To generate the file, use the same loop technique to build a large string of the complete hash/array and then 'syswrite' the string to disk as one operation. To get the hash/array use 'sysread' to get the string back into memory. Then use 'split' to get back your hash/array. Depending on hardware you can get 5 times or more improvement in speed. If using Linux, use the '/dev/shm' Filesystem for saving the string with improved performance ( if your memory is large enough?). Obviously, if the string is too large for memory, then you will need to use a DB and 'BerkeleyDB' is a good choice. Regards...Ed "Well done is better than well said." - Benjamin Franklin	[reply]
Re: Saving a hash/array to a file by LanX (Saint) on Sep 24, 2014 at 01:56 UTC
> the slowest part of the operation is reading it in (yes, I've profiled it). What do you mean with reading it in? How big is your hash? How long does it take to evaluate the literal code of the hash? How many percent of the hash do you need to access in one run of your script? What is "slow" in your eyes? Cheers Rolf _{(addicted to the Perl Programming Language and ☆☆☆☆ :)}	[reply]