If latenecy is your problem, you probably want to patch your kernel to be low latency. Go to Con Kolivas kernel patch , and do groups.google searches for discussion about it. It is widely used for people using their computers for audio work.

But I also agree with sgifford that your should try to streamline your playing of the audio. I would try to put the audio in a format that you can store in memory and write directly to /dev/dsp, the Audio::DSP module works well.