Re: How can I make a hash use less memory

Try a database.

The premise that you need to store millions of items in memory is flawed. No amount of optimization is going to save you when you need such a huge amount of memory.

()-()
 \"/
  `

Comment on Re: How can I make a hash use less memory

Replies are listed 'Best First'.
Re: Re: How can I make a hash use less memory by diotalevi (Canon) on Oct 11, 2002 at 18:10 UTC
And to further expand on ignatz, you can switch to something like DB_File (which is in core perl) or BerkeleyDB and not have to change your code much. You keep your hash - it's just tied to a database. I've used BerkeleyDB to good effect for working with large datasets that didn't fit into memory. Update: you'll probably also want to use something to store your structures as well: Storeable, MLDBM, FreezeThaw, Data::Dumper, etc. You can't just store raw structures but the idea is the same. `__SIG__ printf "You are here %08x\n", unpack "L!", unpack "P4", pack "L!", B:: +svref_2object(sub{})->OUTSIDE` [download]	[reply] [d/l]

Replies are listed 'Best First'.

Re: Re: How can I make a hash use less memory
by diotalevi (Canon) on Oct 11, 2002 at 18:10 UTC

And to further expand on ignatz, you can switch to something like DB_File (which is in core perl) or BerkeleyDB and not have to change your code much. You keep your hash - it's just tied to a database. I've used BerkeleyDB to good effect for working with large datasets that didn't fit into memory.

Update: you'll probably also want to use something to store your structures as well: Storeable, MLDBM, FreezeThaw, Data::Dumper, etc. You can't just store raw structures but the idea is the same.

__SIG__
printf "You are here %08x\n", unpack "L!", unpack "P4", pack "L!", B::
+svref_2object(sub{})->OUTSIDE
[download]

[reply]
[d/l]