I suspect that you are confusing perl's reference with C's pointer. Perl's reference-to-an-array is exactly what its name implies. It really is of no concern to you what it actually contains or how it works. It does not make any sense to use it in an arithmetic expression.

Perl does not have a data type "multi-dimensional-array". Perl simulates it very well with the structure known as an "array-of-arrays". (This is actually an array of references each of which refers to an array.) Perl provides a syntax with multiple indicies to conveniently access individual elements of this structure. You do not know or care where the individual arrays are stored. You have a reference to each one.

I doubt that your data is stored correctly. I cannot be sure without a reasonable sample of your data.