Is is possible to write a script that when fed a
file containing properly formed Unicode text, it will count the
number of words and sentences it contains?
No!
Languages of the world are way too complex. Unicode deals with text
at the character and grapheme level, which is hard enough. It is
silent on what constitutes a word or sentence. It is certainly
possible in many cases to define "words" and "sentences" in a way
appropriate to some particular expected text format in some known
language, but even then there are usually exceptions. Take
choroba's code which satisfies a given spec. Is
Sports.ru
one word or two? Is
какое-то two words, as the code
determines, or just one as Russian linguists would probably contend?
Do all other languages handle hyphenated text similarly? Almost certainly not,
as a general rule. The more text considered, the more edge cases
and ambiguities arise, even within a single language.
I am
slowly but steadily working to handle Thai and Lao text in Perl.
For these languages, sentences are generally delimited by
whitespace, and individual words are not delimited at all in the
text, but instead are delimited by syntactic rules. Code can and
has been written to count individual Thai words, but it is
considerably different and more complicated than counting the
number of character strings between spaces.
