"I have tried packing and unpacking the data to store it as scalars, but this causes performance to be even slower than without the cache."
It turns out that the overhead is onerous - or at least onerous enough that I may as well just not even use the cache.