I don't think it's a UTF8 issue...

You're probably right; it looks more like an UTF-16 (a.k.a. Windows Unicode) issue. See Encode and Encode::Supported.

Try the following in your script:

use Encode qw(from_to);
...
from_to( $_, 'UTF-16LE', 'latin-1') for @sorted;
 print FOUT join("\n", @sorted);
[download]

Change the 'latin-1' to the actual encoding you want.

I think you meant that as a debugging tool? Why else would you do the conversion so late. Here's what the final code should probably look like:

...

my $content;
{
   open(my $fin, '<:encoding(UTF-16le)', $opt_i)
      or die "$0 : cannot read input file \"$opt_i\"\n";
   local $/;
   $content = <$fin>;
}

...

{
   open(my $fout, '>:encoding(iso-latin-1)', $opt_o)
      or die "$0 : cannot write output file \"$opt_o\"\n";
   print $fout join("\n", @sorted);
}
[download]

the output file contains one huge line of

\x{4445}\x{3035}\x{533b}\x{676f}\x{203b}\x{2e34}\x{4e37}\x{6475}\x{736f}\x{203b}\x{6f43}

Seriously, an "od -c" of the output file shows this:

0000000    \   x   {   7   6   4   5   }   \   x   {   6   e   6   5  
+ }
0000020    \   x   {   6   f   7   4   }   \   x   {   4   2   2   0  
+ }
0000040    \   x   {   7   1   7   5   }   \   x   {   6   5   7   5  
+ }
0000060    \   x   {   2   0   3   b   }   \   x   {   5   f   5   2  
+ }
0000100    \   x   {   7   5   5   4   }   \   x   {   6   2   7   2  
+ }
[download]

I actually got my hands on a WinXP machine running ActivePerl and ran the script with this code on XP, and got the above output.

Horrible thing is (for my friend) when I run my original script on XP, it works like I expect - NOT producing the \x00 bytes! I don't understand this, and I can't replicate it, and I can't visit him (10 time zones away) to see WTF is going on.

I just get lines of question marks (joined by \n) as output.

???????????????????????????????????????????????
????????????????????????????????????????????????
????????????????????????????????????????????????
[download]

Try hexdump/od utilities

perl blah ... | hexdump

or
perl blah ... | od -tacx1
[download]

You can get them from
http://gnuwin32.sourceforge.net/packages/hextools.htm
http://gnuwin32.sourceforge.net/packages/coreutils.htm

Have you checked the input file? What exactly happens to the output file after perl writes it? What is the first thing used to open and inspect the file's content? As mentioned in a previous reply, when there is a null byte next to each character byte, it's a sure sign of UTF-16 encoding; if the initial byte is null (or the first two bytes are "\xFE \xFF") it's UTF-16BE; if the second byte is null (or the first two bytes are "\xFF \xFE") it's UTF-16LE.

You can do a simple test on the victim's WinXP box to see if perl is brain-damaged -- e.g.:

#!/usr/bin/perl

$out = "test\xb0 test";
open(O, ">test.txt") or die "$!";
binmode O;
print O $out;
close O;

$s = -s "test.txt";
print "wrote ".length($out)." bytes to test.txt; file size is $s bytes
+\n";
[download]

As for your script, I don't understand why you want to have three copies of the file data in memory (single slurped scalar, array of lines, hash of lines). Why not do it like this?

#!/usr/bin/perl

use strict;
use warnings;
use Getopt::Std;

our ( $opt_i, $opt_o );
my ( $ifh, $ofh );

getopts( 'i:o:' ) and $opt_i and $opt_o 
    or die "Usage:  $0  -i infile -o outfile\n";
warn "reading \"$opt_i\" and writing to \"$opt_o\"\n";

open( $ifh, "<", $opt_i ) or die "$opt_i: $!\n";
open( $ofh, ">", $opt_o ) or die "$opt_o: $!\n";
binmode $ifh;
binmode $ofh;

my %lines;
$lines{$_}++ while (<$ifh>);

my $line_count = 0;
for (sort keys %lines) {
    print $ofh $_;
    $line_count += $lines{$_};
}
close $ofh or die "error on closing output file: $!\n";

warn "read $line_count lines from $opt_i, wrote ".scalar(keys %lines).
    " lines to $opt_o\n";
[download]

Update: after posting, I noticed that the OP code reported input and output line counts, so I added stuff to my version of the script accordingly.

graff:

I was only provided a snippet of the real input file; I don't know if I have the actual first two bytes of the file. I'll ask my friend about that.

What *should* happen to the output file is that another perl script is used to sort the lines by date/time for a different view of the data. My friend has looked at it with Notepad, Notepad2 and a hex editor, and he says it comes out of the undupe script padded with \x00 bytes.

My OSX passes the braindead test, so does my WinXP which is supposedly now set up the way his is. I'll send him that test, thanks!

3 copies - because I'm an idiot, and have RAM to burn. :-) That is a very nice way you did it. That should scale up nicely.

Maybe he has funny stuff in PERL5OPT, try comparing set output and cross reference with perlrun

Byte 0xB0 exists on Windows as well. It may not display the same, but instances of 0xB0 don't suddenly become 0x00 based on the OS being used to read the file.