What I think you want is PDL. You can certainly do arbitrary matrix manipulation with it, and it is quite fast and memory efficient for most tasks I have encountered.