echo -n 'a…b' | perl -pe 's@(.).(.)@$1$2@';

echo may generate a byte stream representing three Unicode characters, but Perl reads it as byte stream, not as Unicode characters. So you are cutting out a byte, not a character, and get back garbage. Also, perl writes out bytes, not Unicode characters.

Tell perl to treat STDIN and STDOUT as Unicode character streams and everything works as expected:

>echo -n 'a…b' | perl -pe 's@(.).(.)@$1$2@'
a▒▒b
>echo -n 'a…b' | perl -CIO -pe 's@(.).(.)@$1$2@'
ab
>perl -v

This is perl 5, version 22, subversion 2 (v5.22.2) built for x86_64-linux-thread-multi

Copyright 1987-2015, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

>

See also -C in perlrun, and the thread any use of 'use locale'?, especially the subthread Re^3: any use of 'use locale'? (source encoding).

Alexander

Thanks for shedding light on this. I'm actually planning to use this in a script, so I see now I need to have:

binmode(STDIN, ':utf8');
binmode(STDOUT, ':utf8');
[download]

at the top of my script.

...Although, I find this works when I collect user input with the construct:

my $user_input = <STDIN>;
[download]

but does not work when I collect it with:

my $stdin = new IO::Handle;
   $stdin->fdopen( fileno( STDIN ), "r" ) || die "Cannot open STDIN";
while ( my $char = $stdin->getc() ) {
}
[download]

and I need to use hippo's suggestion below. In each iteration, $char is a byte. Is there a way to coerce $char to be utf8?

echo -n 'a…b' | perl -pe 'utf8::encode($_); s@(.).(.)@$1$2@';

Ah, you were so close.

Thanks! Counter-intuitive to me, but probably because of my inexperience.

I thought that too when I first came across it. Here's the way I remember it now: Perl has its own internal way of storing the data - don't worry about what it is, just know that it's some special thing. When your code acquires data (from STDIN or another filehandle or a database or a web request) the data has some encoding. To process it you need to decode it from that encoding into the special internal format, so you use decode. Conversely when sending data out of perl (to STDOUT or a database or a web response or ...) you need to encode it back into what the receiving system expects so you use encode. This is why it helps to think of utf-8, utf-16, Latin-1, etc. as encodings (and this is indeed what they are).

All the magic shortcuts you might come across such as encoding layers are performing these operations automatically for you but really it's just decode and encode under the bonnet.

This is a workaround but should remove all non 255-ASCII characters:

use strict;
use warnings;
use utf8;

my $str = 'a…b';
$str =~ s/(.)/ord($1)<256?$1:''/ge;
print "$str\n";
[download]

This works if $str is implicitly provided in the script, but not when it is read from stdin.