Is is possible to write a script that when fed a file containing properly formed Unicode text, it will count the number of words and sentences it contains?

I am slowly but steadily working to handle Thai and Lao text in Perl. For these languages, sentences are generally delimited by whitespace, and individual words are not delimited at all in the text, but instead are delimited by syntactic rules. Code can and has been written to count individual Thai words, but it is considerably different and more complicated than counting the number of character strings between spaces.

Thai and Lao text ... these languages, sentences are generally delimited by whitespace, and individual words are not delimited at all in the text, but instead are delimited by syntactic rules.

So, fair to say that the first requirement to process Unicode 'text'; is to determine the language.

So then the question becomes: given a file of Unicode text; can the language be determined?

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

> can the language be determined?

You know the answer, only with statistical certainty and dependent on the length of the text and the distance of languages.

Hand and finger (en) <=> Hand und Finger (de)

If same script lead to same delimiters can only be answered by someone knowing all 6000 languages of the world.

But already Arabic words should be a problem, maybe less if transcribed. Chinese even more.

see also Word_divider and Word#Word_boundaries

Cheers Rolf

( addicted to the Perl Programming Language)

Again, in the general case, no. There exist texts which are in multiple languages, which may have different syntactic rules. Sometimes the two languages are in separate volumes, or at least separate halves on a volume, but sometimes you'll get the two languages on opposite pages, or in two columns on each page, or even line by line translations. And very occasionally you'll even see line by line translations in more than two langauges. I have a book at home that is tri-lingual Greek/Latin/English, for example.

Define words and sentences,

How about starting with the simplest possible definitions:

1. Words: whitespace delimited sequences of letters.
2. Sentences: sets of words delimited by a full stop.

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Words are usually delimited by punctuation, not only whitespace. Therefore, the following script only counts letters, delimited by non-letters.

#!/usr/bin/perl
use warnings;
use strict;

use open IO => ':utf8', ':std';

my ($words, $sentences);
while (<>) {
    $words++     for m/\p{L}+/g;
    $sentences++ for m/\./g;
}
print "$words $sentences\n";
[download]

Tested on the following text:

Огонь XXII Зимних олимпийских игр в Сочи во второй раз погас в понедельник в Москве, во время этапа эстафеты олимпийского огня. После нескольких безуспешных попыток снова его зажечь, факел был заменен, передает портал Sports.ru.
Казус произошел на Раушской набережной, недалеко от Кремля. Видно, как зрители приветствуют факелоносца, он машет в ответ, и через какое-то время факел гаснет.

Output:

59 5
[download]

لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ

How many sentences are there in these examples?

• He said "I like pie. I also like tickles."
• He said "The pie cost me £2.30."
• I like pie
• Who watches the watchers?

Some will argue that something like "court martial" is a single word, despite having a space in.

And then there are all those pesky non-European languages. Apparently Chinese uses the same space between words as between characters.

So the answer is "probably not", in the general case.