Thanks all for your help. I was silly to manually vivify the array ref. I tried a few variants, but I cannot squeeze out the performance that I want. Maybe the problem can be solved on the writing end:
Is they a good (best!) way to store a hash of variable-length arrays of unsigned integers? Instead of key,value pairs with redundant keys?
-poster