Hashes are great when you want to do fast lookups of data by key, but for inherently indexed datasets, they make very little sense at all, as they do not preserve order.

For matrix operations, an AoA makes much more sense. If pure speed is your criteria and each of your rows have the same number of elements (ditto the columns), then using a single dimension array and using math to convert your 2d indices to 1d indices might prove a little quicker.