Hi!

Have you ever worked with CSound?

Hacking Perl in Nightclubs on perl.com describes how to connect Perl and CSound using OSC (Open Sound Control) messages via TCP/IP.
The CSound Journal also contains two articles on using Perl and CSound, but they rather focus on generating CSound code with Perl:
Perl and Csound - Part I
Perl and Csound - Part II

CSound is a complex audio programming language in itself, but it looks quite flexible to me.

Have fun!