Consider pack/unpack in a pure perl implementation. It may be helpful to set the record delimiter $/ to a reference to a constant integer. That triggers magic which will let you read a file in fixed length chunks.

There is a PDL::Audio module, but I had difficulty with it, seemed to be version problems. PDL in general is a good way to improve the performance of numerical array operations. I'm currently using it to do FFT of data.

Update: Have you looked at Mmap for working on these files? Here's a PDL FFT example with a small enough data set to show results explicitly:

#!/usr/bin/perl -w
use strict;

use PDL;
use PDL::IO::Misc;
use PDL::FFT qw(:Func);

use constant TWOPI=>8*atan2 1,1;

my $data = PDL->new([ map { exp(-$_/10) * cos(TWOPI*3*$_/16) } 0..15 ]
+);

# this is a print routine
print 'Original Data:',$/;
PDL::IO::Misc::wcols($data);

# works in place, modifies $data
realfft($data);

print 'Transformed Data:',$/;
PDL::IO::Misc::wcols($data);

realifft($data);

print 'Restored Data:',$/;
PDL::IO::Misc::wcols($data);
[download]

$ perl  fft.pl
Original Data:
1 
0.346266288866367 
-0.57893006746741 
-0.684426791399269 
-1.23131728120463e-16 
0.56036126234907 
0.38806842947617 
-0.190034968516957 
-0.449328964117222 
-0.155587472885039 
0.260130047511444 
0.307532781193507 
1.65979955538993e-16 
-0.251786545542726 
-0.174370385423124 
0.0853882155497732 
Transformed Data:
0.463281829594583 
0.486027823589927 
0.659813926688149 
4.40456725406594 
0.655773011785699 
0.476109468453563 
0.441528145077408 
0.43061131035946 
0.427856290365133 
-0.275073975558085 
-0.808370936709314 
0.0824047624460787 
0.980794295960617 
0.473606356659868 
0.256625191308709 
0.116127618655707 
Restored Data:
1 
0.346266288866367 
-0.57893006746741 
-0.684426791399269 
-2.02962646689286e-16 
0.56036126234907 
0.38806842947617 
-0.190034968516957 
-0.449328964117222 
-0.155587472885039 
0.260130047511444 
0.307532781193507 
2.02962646689286e-16 
-0.251786545542726 
-0.174370385423124 
0.0853882155497732 

After Compline,
Zaxo