My personal guidelines are:

1. Start with Perl
2. If it's too slow, think about optimizations
3. If there are no obvious optimizations, either incorporate a fast XS module that does the most work, or write in C instead

The default choice of perl is simply because I know Perl much better than C, because I don't like segfaults, because string handling is so cumbersome in C, and because I'm just more productive in Perl.

I've done quite some small scientific computations for my studies, and I only needed C's speed very rarely.

As a side note there's currently a Google Summer of Code project on its way to bring GSL to perl. (GSL is the GNU Scientific Library).

BTW most of the time when you need raw speed it's best to use some well-optimized libraries, like the lapack or atlas libs. Even Matlab can be surprisingly fast when you formulate all your operations as vector operations instead of loops.