Re^3: Perl & Unicode: state of the art?

Words are usually delimited by punctuation, not only whitespace. Therefore, the following script only counts letters, delimited by non-letters.

#!/usr/bin/perl
use warnings;
use strict;

use open IO => ':utf8', ':std';

my ($words, $sentences);
while (<>) {
    $words++     for m/\p{L}+/g;
    $sentences++ for m/\./g;
}
print "$words $sentences\n";
[download]

Tested on the following text:

Огонь XXII Зимних олимпийских игр в Сочи во второй раз погас в понедельник в Москве, во время этапа эстафеты олимпийского огня. После нескольких безуспешных попыток снова его зажечь, факел был заменен, передает портал Sports.ru.
Казус произошел на Раушской набережной, недалеко от Кремля. Видно, как зрители приветствуют факелоносца, он машет в ответ, и через какое-то время факел гаснет.

Output:

59 5
[download]

لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ

Comment on Re^3: Perl & Unicode: state of the art? Select or Download Code

Replies are listed 'Best First'.
Re^4: Perl & Unicode: state of the art? by Jenda (Abbot) on Oct 08, 2013 at 14:53 UTC
Let's assume Czech. Dots are used after numbers to denote ordinals (1. = 1st, 8. = 8th, etc.). Dots are used between numbers in dates (3.9.1975 or 3. září 1975 = September 3rd 1975). Dots are used at the end of abbreviations, though to make things harder some abbreviations are so common that they do not need the dot and if there's an abbreviation at the end of a sentence, you do not double the dot. And of course some sentences end by a question or exclamation mark. Anything that would not take into account the language would fail on any nontrivial text. Even if it did take the language into account, there would be "false positives" and missed sentences. Jenda Enoch was right! Enjoy the last years of Rome.	[reply]
Re^4: Perl & Unicode: state of the art? by BrowserUk (Patriarch) on Oct 07, 2013 at 22:42 UTC
According to that, this: `...................................................................... +...........................................` [download] Contains a lot of sentences. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l]
Re^5: Perl & Unicode: state of the art? by choroba (Cardinal) on Oct 07, 2013 at 22:48 UTC
Sure. Lots of empty sets of words. ;-) لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ	[reply]