file handling or hash is efficient

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi monks,

I am working on script wherein 1 million of records needs to be uploaded from the data base , say in every fifteen mins,and the with respect to the some input values in need to lookup in these records to get a value.

eg: hash key - 3456789,200805061200000
    hash value - 200805061200000,200905061200000,Group1
[download]

so in my script i will get the two key values as input and i need to do a lookup and fetch value information.

So I am confused about the best approach

1> I need to uploade these records to hash from database directly.

2>Keep a flat files with all these records and load hash with these data at the inital part of the script

3> directly read data from file

kindly help to me to select best option.

I am expecting a 2600 * 1500 input values in fifteen minutes and i have a memory of 64gb.

HTML markup added by GrandFather

Comment on file handling or hash is efficient Download Code

Replies are listed 'Best First'.
Re: file handling or hash is efficient by perliff (Monk) on Jun 24, 2009 at 07:40 UTC
Benchmark is your friend here. just try out all these methods a few times and see what goes faster. And you don't have to install the module from cpan, its already part of the distribution. here's a skeleton of one run for any piece of code you want benchmarked. `use strict; use Benchmark; my $t0 = new Benchmark; # ... your code here ... my $t1 = new Benchmark; my $td = timediff($t1, $t0); print "the code took:",timestr($td),"\n";` [download] although, its advisable to run the same code several times, (say a few hundred times ) and then average out the time to get a better estimate of how long the piece of code takes. This can be done directly with the module as well, and is explained in the docs. perliff ---------------------- -with perl on my side	[reply] [d/l]
A reply falls below the community's threshold of quality. You may see it by logging in.