depends on what kind of matrices sp? you're talking about, but if the start getting 'interesting', and you want to do some even more interesting stuff with them, someone once pointed me to the whole bunch of PDL modules.
their code is, AFAIK, made of lovingly handcoded C, so that should give you a decent performance.
never really looked into the wonders that PDL's supposed to offer, but it sure is on my (AU long) to-do list.