work with unicode data and regexps

kamel has asked for the wisdom of the Perl Monks concerning the following question:

i have written a parser which searchs in a file for regexps. it all went fine but then i found out, that some of the data files are encoded in unicode. i tried several things from manuals and books but it just doesn't work. it seems that perl does not use the right encoding, it finds almost no regularexpression and the output is still with the NULs after every character. if i write the same lines in ASCII-File the keywords are found without any problem. i am using windows xp with active state perl 5.8. thanks vor your help.

#test.pl - testfile with the heart of the parser
#open the file to search trough
unless (open FILE, "<../perl/testproto/Protokoll.log") {
    die "Cannot open Keywordfile: $!\n";
}

#search for the regexp in the file
while (<FILE>) {
  chomp;
  if (/.*Cl(.*)/) {
    print $1;
     }
  else {
    print "Keine Treffer.\n";
  }
}
[download]

Part of the data-file: keywords to be found: Clinic, Surgon, Patient Code, ...

S: 07:02:36 ----------------------------------------------------------
+----------
S: 07:02:36 Clinic: Knie Klinik
S: 07:02:36 Surgon: Prof X
S: 07:02:36 Patient Code: Kiro
S: 07:02:36 Size: 30 cm
S: 07:02:36 Weight: 50 kg
S: 07:02:36 Birthdate: Sa 1. Jan 2000
S: 07:02:36 Sex: male
S: 07:02:36 Indication:
[download]

Comment on work with unicode data and regexps Select or Download Code

Replies are listed 'Best First'.
Re: work with unicode data and regexps by allolex (Curate) on Feb 10, 2003 at 09:42 UTC
You might want to try setting your locale to one that supports Unicode: `use locale; use POSIX 'locale_h'; my $loc = "en_EN.utf-8"; setlocale(LC_CTYPE, $loc) or die "Invalid locale $loc";` [download] Or use a UNIX-GNU/Linux command like recode or iconv to convert your input to your character set. Hope that helps. Addendum: I'm not sure this will work on XP, but perldoc perllocale should be able to help you--and there is a Windows version of recode.	[reply] [d/l]
Re: work with unicode data and regexps by dakkar (Hermit) on Feb 11, 2003 at 15:49 UTC
`open FILE,'<:UTF-16','../perl/testproto/Protokoll.log')` or `binmode FILE,':UTF-16'` If you know the encoding of the file, you can tell Perl what it is, and it will act accordingly. Otherwise, it assumes it is 1 byte = 1 character, and fail on several multi-byte (or variable-length) encodings. Note: I wrote `UTF-16` above since you said you got 'NULs after every character'. You should check if it is indeed `UTF-16`, or maybe `UCS-2` (they're not the same, you know...), or even something else. -- dakkar - Mobilis in mobile	[reply] [d/l] [select]