forks and Shared

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi there, Im trying to increase the capacity of the shared variables of my webcrawler as I'm goind to index million of urls and the number of data shared will grow very fast. I'm using forks shared, but I'm looking for something which can keep the more used on memory and the less used on disk, with this in mind I looked for berkeleydb, but forksberkeleydb dont worked the way I was waiting, It always crash when I try to insert hash of hashes or array of hashes, so it's useless for my intention.
Some of you monks here a suggestion to solve my problem?

Thanks

Comment on forks and Shared

Replies are listed 'Best First'.
Re: forks and Shared by perrin (Chancellor) on Nov 10, 2008 at 16:18 UTC
BerkeleyDB is the fastest thing available. I'd suggest using it directly if the forks module is giving you trouble.	[reply]
Re: forks and Shared by arkturuz (Curate) on Nov 10, 2008 at 15:56 UTC
I'm not sure this is the right solution because it's not a database-based but you could try using Cache::Memcached module (needs memcached). (There's also pretty faster version Cache::Memcached::Fast). It supports inserting nested Perl data structures, it's pretty reliable, and it works great for sharing data between processes.	[reply]
Re: forks and Shared by Anonymous Monk on Nov 10, 2008 at 16:02 UTC
MLDBM	[reply]
Re^2: forks and Shared (depth) by tye (Sage) on Nov 10, 2008 at 16:50 UTC
Note that MLDBM can be a pain to use correctly (see the BUGS section). I think DBM::Deep deals better with complex data structures. But it will also probably use more disk space than MLDBM. MLDBM requires deserialization + serialization steps each time you modify an element so I can imagine cases where DBM::Deep could be much faster than MLDBM. - tye	[reply]