in reply to How to count the vocabulary of an author?

You need a stemmer for the given language and corpus of texts by the author. Read the texts, tokenise them into words, stem each word and store the stem in a hash. At the end, count the number of keys in the hash.

map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]

Replies are listed 'Best First'.
Re^2: How to count the vocabulary of an author?
by Bod (Parson) on Jun 12, 2021 at 18:29 UTC

    You make it sound so easy choroba :)

    Having done a few (simpler) things with language, I guess that finding the stem of each word is the trickiest part.

[reply]
      Well, I have a PhD in mathematical linguistics. Stemming was done in the first year ;-)

      map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]
[reply]
[d/l]
        Well, I have a PhD in mathematical linguistics

        Wow! - Genuinely impressive.

        Can I ask your opinion on Hemingway Editor?
        I use it extensively in producing content for our business marketing, blogs, etc. But I have started writing something to perform a similar task but more tailored to our needs. For example, in marketing the ratio of first person to second person pronouns is (thought to be) important. My version makes extensive use of Lingua::EN::Fathom.

        My attempt is not very far developed and I'd love some informed input before I go much further.

[reply]
      There are look-up tables for that.

      And even if they didn't exist you can derive most stems by statistical analysis, at least with the Indo-European languages I know.°

      Good enough for a word count.

      Cheers Rolf
      (addicted to the Perl Programming Language :)
      Wikisyntax for the Monastery

      °) because they have in most cases a fixed stem. I suppose Finnish to be much harder...

[reply]

In Section Seekers of Perl Wisdom