You should use sox. There is no CPAN sox module as far as I can tell, but it wouldn't be too hard to write one.

Sox is a command line tool for manipulating audio files. It has a trim function that can be used to split audio files.

Sox is fast. The fastest perl module for audio that I know of is Audio::Data, but it works on .au files not .wav. You can convert between these formats with sox. But once you have sox you won't need Audio::Data anyway.