The first version of Perl that will support Unicode in regexps will be 5.8 ;--( You can try using 5.7.2 in the meantime but it is a development version.

You will have to specifically match unicode characters (I guess UTF8) for your expression to skip them. A unicode character might (cargo-cult programming warning here): ([\xC0-\xDF].|[\xE0-\xEF]..|[\xF0-\xFF]...) or they might interfere with the regexps matching the tokens.