[...] always matches exactly one character, so the previous \w* has to backtrack and give up one character, in order for the whole match to succeed.

By the way you can write [^\s] simpler as \S.

The problem here is \w and \S can match the same characters, so if you match a sequence of \w and \S, the rules about which matches what are governed by the backtracking rules of the regex engine, not by what your intuition expects.

If you want to match a word, and then want to allow non-word but also non-whitespace characters, you can say (\w+)\s*. The \s* allows empty non-whitespace character sequences too. The regex engine greedily matches as many characters as possible with the \w+, and happily leaves \s* to match the empty string if the following character is a space.