[^[:^punct:],"\s]
[download]

Update: [:punct:] doesn't include any whitespace characters (in my testing) so your request to exclude \s and my \s are likely superfluous, FYI.