in reply to matching unicode blocks with regular expressions

Instead of testing for ranges of codepoints, it's usually safer (and much more readable) to test for the Script property. perlunicode lists the available scripts; judging from your description, I think you need this: 
s/(\p{Han}+?)\((\p{Hiragana}+?)\)/\\ruby{\1}{\2}/g;

[download]

Running your code with this regex yields

This is English text with \ruby{日本語}{にほんご} mixed in. To test multi-furi text: \ruby{繰}{く}り\ruby{返}{かえ}し

Which I hope is correct.

Update: the output above was produced with perl-5.8.8 on Linux, and can be reproduced with perl-5.10.0. I used the script below (the code tags of perlmonks will kill the example input, though): 

use utf8; 
binmode DATA, ':encoding(UTF-8)';
binmode STDOUT, ':encoding(UTF-8)';
while (<DATA>) {
    $_ =~ s/(\p{Han}+?)\((\p{Hiragana}+?)\)/\\ruby{\1}{\2}/g;
   print;
} 

__DATA__
This is English text with &#26085;&#26412;&#35486;(&#12395;&#12411;&#1
+2435;&#12372;) mixed in. To test multi-furi text: &#32368;(&#12367;)&
+#12426;&#36820;(&#12363;&#12360;)&#12375; 
# should lead to \ruby{&#32368;}{&#12367;}&#12426;\ruby{&#36820;}{&#12
+363;&#12360;}&#12375;.

[download]

Replies are listed 'Best First'.
Re^2: matching unicode blocks with regular expressions
by Pomax (Initiate) on Jan 08, 2009 at 21:37 UTC

    That's very curious... using your suggestion still yields no result on my machine, using activestate's activeperl 5.10.0 for x86 windows. I guess I'll see if installing strawberry perl makes a difference =/

    update

    Made no difference, it still won't play nice =(

    2nd update

    It actually does work, but the pl file itself had not saved itself in utf-8 format. hurray for annoying little 'last thing you think of' problems.

    Thanks for the help, moritz!

    3rd update

    actually, it doesn't work. While the inline example you gave works fine (relying on __DATA__), moving the data to a file called "test.txt", saved in utf-8 encoding, and then running the code again on that file instead, fails again.

    use utf8;
open(READ,"test.txt");
@lines = <READ>;
close(READ);
foreach (@lines) {
    $_ =~ s/(\p{Han}+?)\((\p{Hiragana}+?)\)/\\ruby{\1}{\2}/g;
   print;
}
# force file to save as unicode: &#26085;&#26412;&#35486;

    [download]

    text file:

    This is English text with 日本語(にほんご) mixed in. To test multi-furi text: 繰(く)り返(かえ)し should lead to \ruby{繰}{く}り\ruby{返}{かえ}し.

    resulting text:

    This is English text with 日本語(にほんご) mixed in. To test multi-furi text: 繰(く)り返(かえ)し should lead to \ruby{繰}{く}り\ruby{返}{かえ}し.

    ... any ideas? =/

[reply]
[d/l]
      I may be missing the point here (I'm not a windows user either), but in the code snippet you showed in your "3rd update", I didn't see any evidence that the file handle was getting a utf8 IO layer (or alternatively, that the data being read in was getting "decoded" into perl-internal utf8).

      Did you try doing the open like this? 

      open( READ, "<:utf8", "test.txt" );

      [download]
      You should also be setting the utf8 IO layer on your output file handle too: 
      binmode STDOUT, ":utf8";

      [download]
      I gather that your "resulting text" should have been different from the input text, but wasn't. This would be expected if your regex substitutions are based on unicode classes, but perl doesn't know that the strings are perl-internal utf8 unicode.
[reply]
[d/l]
[select]
[reply]
        that's a filename issue, not a file content one - in this case the content is saved in utf-8 encoding, and the filenames are in plain ascii.
[reply]

In Section Seekers of Perl Wisdom