C++ has had regular expressions (in some degree of support) in-language since C++11, which is now ten or eleven years old. Unless you have to use Boost for this, you may just want to fall back to the language-native implementations. As I was looking to remember how to retrieve matches I found the std::regex_iterator entity. And the example in the C++ reference online is close enough to what you're looking for: regex_iterator

#include <regex>
#include <iterator>
#include <iostream>
#include <string>
 
int main()
{
    const std::string s = "Quick brown fox.";
 
    std::regex words_regex("[^\\s]+");
    auto words_begin = 
        std::sregex_iterator(s.begin(), s.end(), words_regex);
    auto words_end = std::sregex_iterator();
 
    std::cout << "Found " 
              << std::distance(words_begin, words_end) 
              << " words:\n";
 
    for (std::sregex_iterator i = words_begin; i != words_end; ++i) {
        std::smatch match = *i;                                       
+          
        std::string match_str = match.str(); 
        std::cout << match_str << '\n';
    }   
}
[download]

As you can see in the example, they're counting as a word anything that doesn't contain whitespace. You probably want to also exclude punctuation. So you would want to enumerate that in the character class. It gets harder when you want to deal with apostrophes, allowing them in words, while excluding single quoted constructs. That gets complicated fast.

G'day sbrothy,

It sounds like you just need "split /\W+/, $string"; although, I don't know what the "C++ Boost" connection is.

$ perl -E 'say for split /\W+/, " THIS is a variable number of words, 
+spaces  and  punctuation. "'

THIS
is
a
variable
number
of
words
spaces
and
punctuation
[download]

Is the string you showed representative of your data? Do you have words with hyphens or apostrophes? Are sentences with leading whitespace normal?

The simplicity of my solution may be invalid. Perhaps you need something closer to:

$ perl -E 'say for split /[ ,.]+/, " THIS is a variable number of word
+s, spaces  and  punctuation. " =~ s/^[ ,.]*//r'
THIS
is
a
variable
number
of
words
spaces
and
punctuation
[download]

Why be negative?

perl -le 'print for " THIS is a variable number of words, spaces  and 
+ punctuation. " =~ /\w+/g'
THIS
is
a
variable
number
of
words
spaces
and
punctuation
[download]

my $text = " THIS is a variable number of words, spaces  and  punctuat
+ion. ";
my @list = split /\W+/,$text;
[download]

So I'm splitting the text whenever I found one or more characters which are not words.

I don't know whether eliminating empty strings at the beginning and the end of the list is relevant, so I'm leaving this as an exercise to the reader (hint: grep $_,@list might not do what you want if you consider numbers as words).

http://unicode.org/reports/tr36/#Canonical_Represenation

https://metacpan.org/pod/Lingua::Stem::En

Wow, thank you all. I'd expected you'd wait and see if I did any work of my own before answering but I completely forgot about this until now. These answers were all very helpful. Again, thank you! :)

This is ofcourse C++ and not perl. Still, for completeness sake:

I ended up using boost::regex_token_iterator.

So in all it's glorious simplicity it ended up being just:

boost::regex re("\\s+");
[download]

Regards.