http://qs1969.pair.com?node_id=11133791


in reply to Re: How to count the vocabulary of an author?
in thread How to count the vocabulary of an author?

This is sort of an amendment, because it struck me that word counting is much harder than it looks. Let's take a look at the text "little boy" in three different contexts:

First: Let's consider a 10 year old boy living in Rhode Island: He knows the meaning of the words "little" and "boy" and he heard of a bomb named "Little Boy" in school. It made the U.S. win some war a long time ago and every year there is a celebration. So, three words?

Second: A 10 year old British girl. She certainly knows the words "little" and "boy", but she never heard of the things that happened in Japan in 1945. Those things are taught at a later age. So, two words?

Third: A 10 year old girl in Japan. She doesn't know a single word of english. Neither the words "little" nor "boy" have any meaning to her. But every time she walks to school, she walks past the Genbaku dome. She asked her parents about it, and now she knows that an awful and terrifying machine named "Little Boy" killed her great-grandparents and destroyed her city. So, one word?

perl -e 'use Crypt::Digest::SHA256 qw[sha256_hex]; print substr(sha256_hex("the Answer To Life, The Universe And Everything"), 6, 2), "\n";'

Replies are listed 'Best First'.
Re^3: How to count the vocabulary of an author?
by Bod (Parson) on Jun 12, 2021 at 18:49 UTC
    because it struck me that word counting is much harder than it looks

    There is another dimension to what constitutes a word and how they can be differentiated programmatically. They require context which takes us into a totally different level of complexity.

    Let's take the seemingly simple word post

    When you read the word you might think of what someone puts on social media, or perhaps the mail that gets delivered to your door. But equally you may imagine the pole in the ground that keeps your fence upright. Perhaps you have been given a new post at work as your role has changed due to the company being able to post a good profit. Of course, at one time we didn't need to worry - but post the advent of computing, we do!