Until japhy's patch is implemented (japhy++ btw), you'll have to use two ranges (you can do that within a single character class). For clarity, I won't use variables, but the literal characters (\x00 .. \x1f) (I know \x00 can be written like \0, but i like consistency)

$text =~ s/[\x00-\x09\x0b-\x1f]+//g; # + for speed
# or
$text =~ tr/\x00-\x09\x0b-\x1f//d; # tr/// for even more speed
[download]

Another solution, but quite a bit slower, would be using a negative look-ahead assertion:
Update (200201191932+0100) danger has a better solution: negative look-behind. This saves a lot of time, because only the characters that match the first range are subject for the assertion.

# OLD:$text =~ s/(?!\n)[^\x00-\x1f]//g;
$text =~ s/[^\x00-\x1f](?<!\n)//g; # (Can't use + now)

# or (if you need to exclude another character,
# let's say \n and \r

# OLD:$text =~ s/(?![\r\n])[^\x00-\x1f]//g;
$text =~ s/[^\x00-\x1f](?<![\r\n])//g;

# Note: it's usually better to avoid \r and \n and use
# literals like \cM and \cJ or \015, \012 or \x0d, \x0a.
[download]

HTH. (Warning: untested code)

2;0 juerd@ouranos:~$ perl -e'undef christmas'
Segmentation fault
2;139 juerd@ouranos:~$
[download]