in reply to Perl & Unicode: state of the art?
Is is possible to write a script that when fed a file containing properly formed Unicode text, it will count the number of words and sentences it contains?No! Languages of the world are way too complex. Unicode deals with text at the character and grapheme level, which is hard enough. It is silent on what constitutes a word or sentence. It is certainly possible in many cases to define "words" and "sentences" in a way appropriate to some particular expected text format in some known language, but even then there are usually exceptions. Take choroba's code which satisfies a given spec. Is Sports.ru one word or two? Is какое-то two words, as the code determines, or just one as Russian linguists would probably contend? Do all other languages handle hyphenated text similarly? Almost certainly not, as a general rule. The more text considered, the more edge cases and ambiguities arise, even within a single language.
I am slowly but steadily working to handle Thai and Lao text in Perl. For these languages, sentences are generally delimited by whitespace, and individual words are not delimited at all in the text, but instead are delimited by syntactic rules. Code can and has been written to count individual Thai words, but it is considerably different and more complicated than counting the number of character strings between spaces.
|
---|
Replies are listed 'Best First'. | |
---|---|
Re^2: Perl & Unicode: state of the art?
by BrowserUk (Patriarch) on Oct 08, 2013 at 00:25 UTC | |
by LanX (Saint) on Oct 08, 2013 at 00:45 UTC | |
by BrowserUk (Patriarch) on Oct 08, 2013 at 02:16 UTC | |
by Discipulus (Canon) on Oct 08, 2013 at 07:32 UTC | |
by DrHyde (Prior) on Oct 08, 2013 at 10:35 UTC |
In Section
Seekers of Perl Wisdom