Maybe PDL is what you're looking for (all the relevant parts are implemented in C/XS).

Using pure Perl for number crunching is probably not a good idea (speed-wise).