Sorry, UTF-8 was a typo.

I've submitted unasnwered requests to technical support for the software that generates the logs that I'm reading to tell me the encoding. I may never know it, but that doesn't mean that I get to give-up.

Using <:encoding(UTF-16) has worked nicely for a couple of months, then suddenly I started having problems (i.e., malformed HI surrogate). I don't care if I have to skip one record, I just don't want Perl to die.

I switched to decode(UTF-16), however, while Perl doesn't die, it now behaves differently. The output seems to have a space between every char, which I think implies that the encoding is wrong, but why does it work with <:encoding.

Here are my two programs that I thought would be the same:

my $file = shift;
my $enc = "UTF-16";
open(FILE, "<:encoding($enc)", $file) || die("Can't: $!");
while ( <FILE> ) {
    print;
}    
close(FILE);
[download]

my $file = shift;
my $enc = "UTF-16";
open(FILE,$file) || die("Can't: $!");
while ( <FILE> ) {
    my $str = decode($enc,$_);
    print encode($enc,$str);
}    
close(FILE);
[download]

The first technique worked well for a couple of months, but now I'm getting some new chars on which it dies. The second one doesn't die, but I can't regex an of the text. Any other thoughts?

I appreciate your help thus far. My knowledge is obviously limited with regards to encoding. Any help is much appreciated. Thanks!

The first program is buggy. You decode without ever encoding. You'd get a "wide character" warning for some inputs if you had warnings on.

The second program is also buggy, but for a different reason. You presume a line ends at byte 0x10, but that's not true.

And they output differently. The first outputs a mix of iso-latin-1 and UTF-8. The second program outputs UTF-16le or UTF-16be, probably the latter.

use strict;
use warnings;

use open ':std', ':locale';

my $file = shift;
my $enc = "UTF-16";
open(my $fh, "<:encoding($enc)", $file) || die("Can't: $!");
while ( <$fh> ) {
    print;
}
[download]

use strict;
use warnings;

use open ':std', ':locale';

my $file = shift;
my $enc = "UTF-16";
my $file = do {
   open(my $fh, "<:raw", $file) || die("Can't: $!");
   local $/; 
   <$fh>
};
my $str = decode('UTF-16', $_);
print $str;
[download]

The output seems to have a space between every char

That's usually a sign of UTF-16le/UTF-16be/UCS-2le/UCS-2be being treated as ASCII or a derivative like iso-latin-1 or UTF-8.