Re^2: matching unicode blocks with regular expressions

That's very curious... using your suggestion still yields no result on my machine, using activestate's activeperl 5.10.0 for x86 windows. I guess I'll see if installing strawberry perl makes a difference =/

update

Made no difference, it still won't play nice =(

2nd update

It actually does work, but the pl file itself had not saved itself in utf-8 format. hurray for annoying little 'last thing you think of' problems.

Thanks for the help, moritz!

3rd update

actually, it doesn't work. While the inline example you gave works fine (relying on __DATA__), moving the data to a file called "test.txt", saved in utf-8 encoding, and then running the code again on that file instead, fails again.

use utf8;
open(READ,"test.txt");
@lines = <READ>;
close(READ);
foreach (@lines) {
    $_ =~ s/(\p{Han}+?)\((\p{Hiragana}+?)\)/\\ruby{\1}{\2}/g;
   print;
}
# force file to save as unicode: &#26085;&#26412;&#35486;
[download]

text file:

This is English text with 日本語(にほんご) mixed in. To test multi-furi text: 繰(く)り返(かえ)し should lead to \ruby{繰}{く}り\ruby{返}{かえ}し.

resulting text:

This is English text with 日本語(にほんご) mixed in. To test multi-furi text: 繰(く)り返(かえ)し should lead to \ruby{繰}{く}り\ruby{返}{かえ}し.

... any ideas? =/

Comment on Re^2: matching unicode blocks with regular expressions Download Code

Replies are listed 'Best First'.
Re^3: matching unicode blocks with regular expressions by graff (Chancellor) on Jan 09, 2009 at 03:17 UTC
I may be missing the point here (I'm not a windows user either), but in the code snippet you showed in your "3rd update", I didn't see any evidence that the file handle was getting a utf8 IO layer (or alternatively, that the data being read in was getting "decoded" into perl-internal utf8). Did you try doing the open like this? `open( READ, "<:utf8", "test.txt" );` [download] You should also be setting the utf8 IO layer on your output file handle too: `binmode STDOUT, ":utf8";` [download] I gather that your "resulting text" should have been different from the input text, but wasn't. This would be expected if your regex substitutions are based on unicode classes, but perl doesn't know that the strings are perl-internal utf8 unicode.	[reply] [d/l] [select]
Re^3: matching unicode blocks with regular expressions by zentara (Cardinal) on Jan 08, 2009 at 21:48 UTC
I don't use Windows, but Isn't it's standard encoding UTF-16? Not UTF-8. Cannot copy files when filenames got French accentuated character might be helpful. I'm not really a human, but I play one on earth Remember How Lucky You Are	[reply]
Re^4: matching unicode blocks with regular expressions by Pomax (Initiate) on Jan 08, 2009 at 21:59 UTC
that's a filename issue, not a file content one - in this case the content is saved in utf-8 encoding, and the filenames are in plain ascii.	[reply]