A problem with your current regular expression is that [^" ]+ does not allow spaces.

This fails for example for the following string, which I think should be valid:

Corion says "hello sarathy"
[download]

If you want to stay with your approach of matching the tokens (words or quoted parts), I suggest reading perlre, especially on alternation.

For such an approach, I would restate the problem as Match every token that starts with a letter and consists only of letters, or starts with a double-quote and consists of non-double-quotes..