I believe the backreferences don't work inside character classes. But you can use them inside negative lookahead assertions, thus:

/^(\w)(?!.{0,4}\1)(\w)\2(?!.{0,2}\2)(\w)(?!.{0,1}\3)(\w)(?!\4)(\w)\s/;
[download]

That is, find one letter and make sure it doesn't recur in the rest of the pattern. Then find the next, and make sure it's repeated immediately after, but not later on. And so on.