in reply to UCS2 Internationalization file parsing
Take a look at the core Encode and Encode::Supported modules, specifically the encode and decode commands. The PerlIO::encoding module calls those commands internally.
Am I going to need to swap the bytes on every character I read in?
No way. Encode is set up to handle both Big-Endian and Little-Endian. You're having a problem with the Byte Order Mark because you specify an encoding of ucs2, which is an alias for the big-endian format UCS-2BE, but your BOM indicates that the data from the stream is in little-endian format. Perl doesn't like that, and it complains.
Here's a script which illustrates that what you want to do is possible:
#!/usr/bin/perl use strict; use warnings; binmode( STDOUT, ':utf8' ); my $big_endian_BOM = pack( 'n', 0xFEFF ); my $smiley = pack( 'n', 0x263A ); my $ack = pack( 'n', 0x0006 ); my $newline = pack( 'n', 0x000A ); my $data = $big_endian_BOM; $data .= "$smiley$newline$ack$smiley$newline$ack"; open( my $fh, '<:encoding(ucs2)', \$data ) or die $!; local $/ = $ack; while (<$fh>) { chomp; print; }
It prints two smileys, each on its own line. No errors or warnings.
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^2: UCS2 Internationalization file parsing
by sfinster (Acolyte) on Mar 29, 2006 at 16:46 UTC |