A matrix is a two-dimensional ordered data structure. An array of arrayrefs is a simple way to represent this in Perl. Read perllol, perldsc. Hashes don't help you here.
If you want to do calculation with matrices, use a CPAN module such as Math::GSL::Matrix.