Ok, problem solved. In the end I did use a (very simplistic) tokenizer. I didn't post source code with my original question because I'm actually working in PHP (but with Perl compatible regular expressions). I wanted a regex-only solution, but in the end tokenizer+regex seemed to be the shortest and most robust solution.

If you follow this link you'll see a more detailed explanation and there's a link to the source.

http://greghurrell.net/wp/2005/06/27/autolink-plug-in/