You mentioned that you store most of the data in hashes. Why not try only loading data into the hash when it is asked for, using the hash as a cache? Then you get the speed of the hash without loading all of the data into the hash, when you may only need part of it.

An example cribbed more or less from page 609 of Programming Perl, 3d edition:

#!/usr/bin/perl -w
use strict;
use warnings;

sub get_data {
  # here we simulate the expensive operation 
  # of loading data into the hash
  my $idx = shift;
  print "retrieving data for $idx...\n";
  return "value for $idx";
}
my $result;
my %cache;
foreach (1,2,4,16,32,2,64,1,2) {
  # here we are asking for values from the cache
  # (hash), and if they aren't there, we get them
  # from the file.
  $result = $cache{$_} ||= get_data($_);
  print "$result\n";
}

[download]

The output from this program is:

retrieving data for 1...
value for 1
retrieving data for 2...
value for 2
retrieving data for 4...
value for 4
retrieving data for 16...
value for 16
retrieving data for 32...
value for 32
value for 2
retrieving data for 64...
value for 64
value for 1
value for 2

[download]
As you can see, after the first time we ask for an element, we no longer have to go back to the file to get the data. Neat, eh? I hope that helps you. -zeno

In reply to Re: Data. Lots of Data. by zeno
in thread Data. Lots of Data. by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.