in reply to Re: Mysterious Whitespaces between each character in a file (hack for 5.6.x)
in thread Mysterious Whitespaces between each character in a file

$/="\n\0"; will fail if the file contains character U+0Axx followed by U+yy00 (for any values "xx" and "yy").

Also, you should replace characters outside iso-latin-1 with some fixed character (such as "?") rather than some random character.

This fixes both problems: 

local $/ = "\x0A\x00";
for (
    my $line = '';
    defined( $_ = <> );
    $line = ''
) {
    $line .= $_;
    redo if length($line) % 2 != 0;

    print
        pack 'C*',
        map { $_ <= 0xFF ? $_ : '?' }
        unpack 'v*',
        $line;

    #   -or-
    # print utf16le($line)->latin1();
}

[download]

(Assumes each file in @ARGV is properly formed, i.e. contain an even number of bytes.)

Replies are listed 'Best First'.
Re^3: Mysterious Whitespaces between each character in a file (hack for 5.6.x)
by almut (Canon) on Oct 08, 2009 at 17:08 UTC
    $/="\n\0"; will fail if the file contains character U+0Axx followed by U+yy00

    This is correct (same holds for "\x0A\x00", btw).  However, as U+0Axx is Gurmukhi/Gujarati, this is rather unlikely to happen in the OP's case... (Also, a characteristic of a "crude hack" (as I called it) is, that it would work in most practical cases, but isn't failsafe, theoretically).

[reply]
[d/l]
[select]

      same holds for "\x0A\x00", btw

      Yes. It's not the (purely æsthetic) switch from \n to \x0A that solves the problem, it's the redo.

      a characteristic of a "crude hack" (as I called it) is, that it would work in most practical cases

      Yes, which is why I wouldn't have mentioned it if I hadn't already been replying about map $_ & 0xff, being worthless (harmful?).

[reply]
[d/l]
[select]
        It's not the (purely ęsthetic) switch from \n to \x0A that solves the problem, it's the redo.

        I'm aware of that, but the casual reader might not be, and misinterpret your post to mean that the fix for the U+0Axx-U+yy00 issue is the change from "\n\0" to "\x0A\x00" — which is why I thought it's worth pointing it out...

        ...about map $_ & 0xff, being worthless (harmful?)

        ...harmful only if there actually are characters outside of the latin1 range (which didn't seem to be the case with the input presented by the OP). And even then the "harm" is just an occasional incorrect arbitrary character instead of a "?" (so why worry, with a crude hack?).

        As to being worthless, it at least makes explicit what the intention is, and avoids the warning "Character in 'C' format wrapped in pack" with newer versions of Perl when the value is greater than 0xff (and when warnings are enabled). You're right however in that with 5.6.1 it's worthless (strictly speaking), as 5.6.1 does not issue a warning in this case, but rather silently truncates the value.

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom