Re^4: Decoding bad UTF-16

Sorry, UTF-8 was a typo.

I've submitted unasnwered requests to technical support for the software that generates the logs that I'm reading to tell me the encoding. I may never know it, but that doesn't mean that I get to give-up.

Using <:encoding(UTF-16) has worked nicely for a couple of months, then suddenly I started having problems (i.e., malformed HI surrogate). I don't care if I have to skip one record, I just don't want Perl to die.

I switched to decode(UTF-16), however, while Perl doesn't die, it now behaves differently. The output seems to have a space between every char, which I think implies that the encoding is wrong, but why does it work with <:encoding.

Here are my two programs that I thought would be the same:

my $file = shift;
my $enc = "UTF-16";
open(FILE, "<:encoding($enc)", $file) || die("Can't: $!");
while ( <FILE> ) {
    print;
}    
close(FILE);
[download]

my $file = shift;
my $enc = "UTF-16";
open(FILE,$file) || die("Can't: $!");
while ( <FILE> ) {
    my $str = decode($enc,$_);
    print encode($enc,$str);
}    
close(FILE);
[download]

The first technique worked well for a couple of months, but now I'm getting some new chars on which it dies. The second one doesn't die, but I can't regex an of the text. Any other thoughts?

I appreciate your help thus far. My knowledge is obviously limited with regards to encoding. Any help is much appreciated. Thanks!

Comment on Re^4: Decoding bad UTF-16 Select or Download Code

Replies are listed 'Best First'.
Re^5: Decoding bad UTF-16 by ikegami (Patriarch) on Sep 29, 2008 at 21:07 UTC
The first program is buggy. You decode without ever encoding. You'd get a "wide character" warning for some inputs if you had warnings on. The second program is also buggy, but for a different reason. You presume a line ends at byte 0x10, but that's not true. And they output differently. The first outputs a mix of iso-latin-1 and UTF-8. The second program outputs UTF-16le or UTF-16be, probably the latter. `use strict; use warnings; use open ':std', ':locale'; my $file = shift; my $enc = "UTF-16"; open(my $fh, "<:encoding($enc)", $file) \|\| die("Can't: $!"); while ( <$fh> ) { print; }` [download] `use strict; use warnings; use open ':std', ':locale'; my $file = shift; my $enc = "UTF-16"; my $file = do { open(my $fh, "<:raw", $file) \|\| die("Can't: $!"); local $/; <$fh> }; my $str = decode('UTF-16', $_); print $str;` [download] The output seems to have a space between every char That's usually a sign of UTF-16le/UTF-16be/UCS-2le/UCS-2be being treated as ASCII or a derivative like iso-latin-1 or UTF-8.	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^5: Decoding bad UTF-16
by ikegami (Patriarch) on Sep 29, 2008 at 21:07 UTC

The first program is buggy. You decode without ever encoding. You'd get a "wide character" warning for some inputs if you had warnings on.

The second program is also buggy, but for a different reason. You presume a line ends at byte 0x10, but that's not true.

And they output differently. The first outputs a mix of iso-latin-1 and UTF-8. The second program outputs UTF-16le or UTF-16be, probably the latter.

use strict;
use warnings;

use open ':std', ':locale';

my $file = shift;
my $enc = "UTF-16";
open(my $fh, "<:encoding($enc)", $file) || die("Can't: $!");
while ( <$fh> ) {
    print;
}
[download]

use strict;
use warnings;

use open ':std', ':locale';

my $file = shift;
my $enc = "UTF-16";
my $file = do {
   open(my $fh, "<:raw", $file) || die("Can't: $!");
   local $/; 
   <$fh>
};
my $str = decode('UTF-16', $_);
print $str;
[download]

The output seems to have a space between every char

That's usually a sign of UTF-16le/UTF-16be/UCS-2le/UCS-2be being treated as ASCII or a derivative like iso-latin-1 or UTF-8.

[reply]
[d/l]
[select]