I do not understand your application well enough to offer a solution, but I do have several comments.

Awkward perl code is usually a symptom of a poor choice of data structures. Your code makes extensive use of "symbolic reference." Most text books consider this very poor practice. (It is not even allowed under "use strict".) My first guess would be to change your matrix to hash-of-hashes.