in reply to UCS2 Internationalization file parsing

Take a look at the core Encode and Encode::Supported modules, specifically the encode and decode commands. The PerlIO::encoding module calls those commands internally.

Am I going to need to swap the bytes on every character I read in?

No way. Encode is set up to handle both Big-Endian and Little-Endian. You're having a problem with the Byte Order Mark because you specify an encoding of ucs2, which is an alias for the big-endian format UCS-2BE, but your BOM indicates that the data from the stream is in little-endian format. Perl doesn't like that, and it complains.

Here's a script which illustrates that what you want to do is possible:

#!/usr/bin/perl use strict; use warnings; binmode( STDOUT, ':utf8' ); my $big_endian_BOM = pack( 'n', 0xFEFF ); my $smiley = pack( 'n', 0x263A ); my $ack = pack( 'n', 0x0006 ); my $newline = pack( 'n', 0x000A ); my $data = $big_endian_BOM; $data .= "$smiley$newline$ack$smiley$newline$ack"; open( my $fh, '<:encoding(ucs2)', \$data ) or die $!; local $/ = $ack; while (<$fh>) { chomp; print; }

It prints two smileys, each on its own line. No errors or warnings.

--
Marvin Humphrey
Rectangular Research ― http://www.rectangular.com

Replies are listed 'Best First'.
Re^2: UCS2 Internationalization file parsing
by sfinster (Acolyte) on Mar 29, 2006 at 16:46 UTC
    If I run your smiley script on my Windows 2000 machine and get garbage (looks like it might be extended ASCII), what do I need to change?

    I have "Central Europe" and "Western Europe and United States" enabled in Control Panel->Regional Options. Locale is English (United States).

    In my script if I try to encode what I've read in (back to original question) I get a "Wide character in input" error. Print doesn't show me the correct data even though I've used binmode to set STDOUT.

    Are there other environment settings I need to be making?