I just used Strawberry Perl 5.12.3.0 portable edition and was able to install Math::FFT directly from CPAN without any issues.

What distribution (ActivePerl, Strawberry, etc.) and version (5.12.30, 5.16, etc.) of Perl are you using?