Just use another character class than \s. Non-word, \W, might be what you want.

If all else fails you can make a custom character class with square brackets. See perlre, the regex doc.