in reply to Large files tied by BerkeleyDB with MLDBM

First, the MLDBM tie mechanism serializes the data structure at the time you make the assignment. I.e., your

my @seqRelation; $hash{'key'} = \@seqRelation;

just stores an empty array.  Later changes to @seqRelation will not automatically be updated in the tied storage.  In other words, you'd need to assign it after you've pushed all your data onto it.

Secondly, note that what you store under a single key will be stored as one string.  So I'm not really sure what you're hoping to achieve with this approach, memory-wise.  If your data doesn't fit into a single data structure in memory, it won't fit into @seqRelation either, and serializing this huge data structure would additionally require quite a lot of memory...

What might work - if you want to stick with BerkeleyDB - is to store each record separately, e.g. using the line number as the key, which you could then use as a pseudo array index for retrieval.

Replies are listed 'Best First'.
Re^2: Large files tied by BerkeleyDB with MLDBM
by Anonymous Monk on May 13, 2011 at 20:29 UTC
    Thank you for your reply. I have realized this problem. Anyway I found perl is not good at manipulating large datasets. Now I have converted my data into string before restore and convert it back when retrieving it.