Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine

Re^2: How to count the vocabulary of an author?

by Bod (Curate)
on Jun 12, 2021 at 18:29 UTC ( #11133807=note: print w/replies, xml ) Need Help??

in reply to Re: How to count the vocabulary of an author?
in thread How to count the vocabulary of an author?

You make it sound so easy choroba :)

Having done a few (simpler) things with language, I guess that finding the stem of each word is the trickiest part.

  • Comment on Re^2: How to count the vocabulary of an author?

Replies are listed 'Best First'.
Re^3: How to count the vocabulary of an author?
by choroba (Archbishop) on Jun 12, 2021 at 18:33 UTC
    Well, I have a PhD in mathematical linguistics. Stemming was done in the first year ;-)

    map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]
      Well, I have a PhD in mathematical linguistics

      Wow! - Genuinely impressive.

      Can I ask your opinion on Hemingway Editor?
      I use it extensively in producing content for our business marketing, blogs, etc. But I have started writing something to perform a similar task but more tailored to our needs. For example, in marketing the ratio of first person to second person pronouns is (thought to be) important. My version makes extensive use of Lingua::EN::Fathom.

      My attempt is not very far developed and I'd love some informed input before I go much further.

        In fact, the idea is craftily clever. Their stemmer and parser can only stem and parse simple sentences, so if it can't process the sentence with a sufficiently high certainty, they flag it as too complex :-)

        I don't know what technology they use in the editor. Also, I quit academia almost ten years ago, so things might have moved a bit since I worked on similar stuff.

        But generally, English is one of the easier languages to process. Its morphology is simple (almost no declension, simple conjugation) and the training data for statistical methods are huge.

        map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]
Re^3: How to count the vocabulary of an author?
by LanX (Sage) on Jun 12, 2021 at 21:02 UTC
    There are look-up tables for that.

    And even if they didn't exist you can derive most stems by statistical analysis, at least with the Indo-European languages I know.

    Good enough for a word count.

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery

    ) because they have in most cases a fixed stem. I suppose Finnish to be much harder...

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11133807]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (3)
As of 2022-08-08 14:13 GMT
Find Nodes?
    Voting Booth?

    No recent polls found