"I have tried packing and unpacking the data to store it as scalars, but this causes performance to be even slower than without the cache." It turns out that the overhead is onerous - or at least onerous enough that I may as well just not even use the cache.

Then you are doing it wrong.

There is no way that unpacking six values from a packed string should be slower than reading those same six values from disk. And querying them from a DB will be far slower still.

I guess it is time you started posting some of your code and let us see what you are doing wrong.

Here is the code to encache/decache. I'm not sure what the problem could be, unless I shouldn't be lexically scoping the rows or using A10 instead of something else in the pack/unpack.

sub to_cache {

    my $ticker = shift;
    my $data = shift;

    my @list;

    foreach(@$data) {
        push @list, pack("A10FFFFL", @$_);
    }

    $data_cache{$ticker} = \@list;
}

sub from_cache {

    my $ticker = shift;
    my $data = $data_cache{$ticker};
    my @rval;

    foreach (@$data) {
       my @row = unpack("A10FFFFL", $_);
       push @rval, \@row;
    }

    return \@rval;
}
[download]

What value or range of values would you see if you added:

sub to_cache {

    my $ticker = shift;
    my $data = shift;

    print scalar @$data; ############# What would be printed?

    my @list;

    foreach(@$data) {
        push @list, pack("A10FFFFL", @$_);
    }

    $data_cache{$ticker} = \@list;
}
[download]

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.