Is is possible to write a script that when fed a file containing properly formed Unicode text, it will count the number of words and sentences it contains?
No! Languages of the world are way too complex. Unicode deals with text at the character and grapheme level, which is hard enough. It is silent on what constitutes a word or sentence. It is certainly possible in many cases to define "words" and "sentences" in a way appropriate to some particular expected text format in some known language, but even then there are usually exceptions. Take choroba's code which satisfies a given spec. Is Sports.ru one word or two? Is какое-то two words, as the code determines, or just one as Russian linguists would probably contend? Do all other languages handle hyphenated text similarly? Almost certainly not, as a general rule. The more text considered, the more edge cases and ambiguities arise, even within a single language.

I am slowly but steadily working to handle Thai and Lao text in Perl. For these languages, sentences are generally delimited by whitespace, and individual words are not delimited at all in the text, but instead are delimited by syntactic rules. Code can and has been written to count individual Thai words, but it is considerably different and more complicated than counting the number of character strings between spaces.

In reply to Re: Perl & Unicode: state of the art? by farang
in thread Perl & Unicode: state of the art? by BrowserUk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.