Re: Problem with a text-parsing regex

Rather than trying to roll your own regex (and depending on what you're trying to do with this next) you probably want to look at CPAN and search for NLP modules (Natural Language Processing) instead. Those are likely going to do what you want WRT removing not-words as well as being able to give you more info about the words/tokens it extracts.

The cake is a lie.
The cake is a lie.
The cake is a lie.

Comment on Re: Problem with a text-parsing regex

Replies are listed 'Best First'.
Re^2: Problem with a text-parsing regex by ibm1620 (Hermit) on May 07, 2022 at 21:00 UTC
Oddly enough, I found almost nothing of use in CPAN related to NLP! That's very surprising. (See, for example, https://metacpan.org/pod/Text::NLP) In any event, there's more idiosyncratic processing of the words and surrounding punctuation, so it's doubtful that any CPAN module would exactly fit my needs.	[reply]
Re^3: Problem with a text-parsing regex by tangent (Parson) on May 07, 2022 at 23:44 UTC
The NLP modules are generally found in the "Lingua" namespace - most of the various part-of-speech taggers, lemmatizers, stemmers etc. have a Perl implementation. Some examples: Lingua::EN::Tagger Lingua::TreeTagger Lingua::FreeLing3 Lingua::EN::SENNA Lingua::CollinsParser Lingua::BrillTagger If you are going to do further NLP type analysis they are worth looking into.	[reply]