Have you looked at Math::Matrix? Don't know how fast it is, but they've done all the "work" for you. Even if it's not suitable for what you want, it may inspire you with some other ideas.