Re: Converting utf-8 to base64 and back

Because you switched encoding and decoding to/from UTF-8:

#!/usr/bin/perl
use strict;
use warnings;
use utf8 ;

use open IO => ':encoding(UTF-8)', ':std';

use Encode;
use Data::Dump qw/dd pp/;
use Devel::Peek;
use MIME::Base64 ;

my $str ='&#345;';

warn "\n*** orig str='$str'";
Dump $str;

my $bytes = Encode::encode_utf8($str);
warn "\n*** encode utf8='$bytes'";
Dump $bytes;

our $encoded = MIME::Base64::encode_base64($bytes);
warn "\n*** encode_b64 encoded='$encoded'";
Dump $encoded;


warn "\n*** decode_b64";
our $decoded = MIME::Base64::decode_base64($encoded);
Dump $decoded;

warn "\n*** encode_utf8";
$decoded = Encode::decode_utf8($decoded);
Dump $decoded;

warn "\n*** _utf8_on (NO-OP)";
Encode::_utf8_on($decoded);
Dump $decoded;
[download]

($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord
}map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,
[download]

Comment on Re: Converting utf-8 to base64 and back Select or Download Code

Replies are listed 'Best First'.
Re^2: Converting utf-8 to base64 and back by LanX (Saint) on Jan 22, 2017 at 00:49 UTC
Thanks I just realized it. (was compiling an answer but YOU were faster ;) It's confusing encode_base64 is producing a base64 string but encode_utf8 is producing an octet-string (i.e. bytes oriented) while decode_utf8 is producing an utf8 character string from an octet string so encode into base64 but encode from utf8! update and my initial example didn't fail because Ä is not a wide character (code-point <256), while ř (code-point 345) is. Cheers Rolf _{(addicted to the Perl Programming Language and ☆☆☆☆ :) Je suis Charlie!} update from Encode#Basic-methods > CAVEAT: When you run $octets = encode("utf8", $string) , then $octets might not be equal to $string. Though both contain the same data, the UTF8 flag for $octets is always off. When you encode anything, the UTF8 flag on the result is always off, even when it contains a completely valid utf8 string. See The UTF8 flag below. so it's even more complicated encode_utf8 does convert into utf8 but w/o flag ..	[reply]

Replies are listed 'Best First'.

Re^2: Converting utf-8 to base64 and back
by LanX (Saint) on Jan 22, 2017 at 00:49 UTC

YOU

It's confusing

encode_base64 is producing a base64 string
but encode_utf8 is producing an octet-string (i.e. bytes oriented)
while decode_utf8 is producing an utf8 character string from an octet string

into

from

update

and my initial example didn't fail because Ä is not a wide character (code-point <256), while ř (code-point 345) is.

Cheers Rolf
_{(addicted to the Perl Programming Language and ☆☆☆☆ :)

Je suis Charlie!}

update

from Encode#Basic-methods

> CAVEAT: When you run $octets = encode("utf8", $string) , then $octets might not be equal to $string. Though both contain the same data, the UTF8 flag for $octets is always off. When you encode anything, the UTF8 flag on the result is always off, even when it contains a completely valid utf8 string. See The UTF8 flag below.

so it's even more complicated encode_utf8 does convert into utf8 but w/o flag ..

[reply]