Is that not what was wanted?

Sorry, I thought you were quoting my post (I used /\s*\b\s*/ first, but then changed it). The only problem with the code there is that I saw it handling this strangely:

a := "test";   # a|:= "|test|";
[download]

In short, any consecutive "token" characters, even separated by spaces, were caught up as the same thing, and I figured that was undesirable. The solution then seemed to be to start from breaking on spaces, and then break upon word boundaries. That seemed to get the best result, but still won't catch things like a trailing "; as separate items.

There's always going to be special cases like this though when using a regex to do the work of a real parser...