I've done some audio programming, but I skipped on the hard parts and just used the fmod library. This was on Windows, but likely you'd be able to write an XS wrapper to make fmod do your bidding. I just loaded the fmod.dll via Win32::API, which isn't that portable.