How to deal with a mass HoA

jiwei800715 has asked for the wisdom of the Perl Monks concerning the following question:

I have a program which collect data and save it in binary file, It can build a index file IDX.txt with 5 columns ( UID,SecID,Date,Time,address), address is a pointer to data. That program's work flow as below /1. load index as a HoA/ /2. waitting for task/ when get a task, it will do /A. search index HoA to find a new entry and new UID / /B. write data and update index HoA / /C. sort index HoA and Save it to IDX.txt file/ As time goes by, I find the IDX.txt is larger than 100M It is very slow when I load IDX.txt and search HoA (grep, sort ). It is necessary to hold Index HoA when program is running, Is there any better way to go through this problem? I am wondering whether Cache module is a better solution, maybe any other module? thanks

Comment on How to deal with a mass HoA

Replies are listed 'Best First'.
Re: How to deal with a mass HoA by wfsp (Abbot) on Mar 24, 2009 at 13:53 UTC
It may be worth having a look at a disk based hash like DBM::Deep. Writeup Formatting Tips is also worth a look. :-)	[reply]
Re: How to deal with a mass HoA by almut (Canon) on Mar 24, 2009 at 14:09 UTC
It is very slow when I load IDX.txt and search HoA (grep, sort ). Using `grep` and `sort` in combination with a hash sounds like you might be doing something in a suboptimal way. Could you post some code so we can see what exactly you're doing? Just a thought...	[reply] [d/l] [select]
Re: How to deal with a mass HoA by zentara (Cardinal) on Mar 25, 2009 at 12:27 UTC
initiate fusion :-) I'm not really a human, but I play one on earth My Petition to the Great Cosmic Conciousness	[reply]