Re: what is the maximum elements storage in a Hash or an array variable

I know this is a really old post ... but ... I just ran a script that created 20 million 192 byte tokens and wrote them to a file as well as adding them to a hash as keys. I'm happy to say there were no duplicates and my Strawberry Perl had no problem handling a hash with 20 million keys, although it sure did slow down my desktop computer while it was running.

Perl rawks ... !

Unfortunately, the resulting 3.8 Gb file was too big for any of my text editors to open, so I rewrote it to create multiple smaller files. If only Microsoft stuff had been created by Larry Wall ... just sayin'.

Comment on Re: what is the maximum elements storage in a Hash or an array variable

Replies are listed 'Best First'.
Re^2: what is the maximum elements storage in a Hash or an array variable by QM (Parson) on Oct 11, 2017 at 09:08 UTC
You may be interested in DBM::Deep, which can map a memory hash to disk. It can also write files much larger than 4GB, on 64bit systems (see Large File Support). It's also slower than a pure in memory hash, but the on-disk hash is persistent, and may perform better than a disk-swapping context. (I'm not associated with DBM::Deep, but I used it extensively some years ago, and it saved a lot of time and trouble. So I always try to give back and suggest it where appropriate.) -QM -- Quantum Mechanics: The dreams stuff is made of	[reply]

Replies are listed 'Best First'.

Re^2: what is the maximum elements storage in a Hash or an array variable
by QM (Parson) on Oct 11, 2017 at 09:08 UTC

DBM::Deep

Large File Support

It's also slower than a pure in memory hash, but the on-disk hash is persistent, and may perform better than a disk-swapping context.

(I'm not associated with DBM::Deep, but I used it extensively some years ago, and it saved a lot of time and trouble. So I always try to give back and suggest it where appropriate.)

-QM
--
Quantum Mechanics: The dreams stuff is made of

[reply]