in reply to Re^2: Perl strings questions
in thread Perl strings questions

Thanks, that clarifies some things. Yet, the python code does not mix text and binary.

As far as I read the code, the binary stuff is BASE64 encoded. Well, yes, unfortunately "encoding" is used for a lot of things.

Let me try to explain the difference:

Binary data will in most cases contain bytes in the range 128..255. Their UTF-8 encoding is not equal to their byte value. If you encode such bytes in UTF-8, it is like Perl interpreting their byte values as code points: Unicode has code points in that range with (not so) surprising similarity to ISO-8859-1. The code point for ö is U+00F6, but its UTF-8 encoding has two bytes X'C3B6'. So, if you encode binary data in UTF-8, the result is different, the process is deterministic and it is reversible.

However, it depends on the receiving side to do a decoding of an UTF-8 stream into binary data and not into a unicode string. Perl happens to do that (because, as you wrote, it makes no difference), but not many other languages do. In general, you can not decode an UTF-8 stream into binary if it contains one or more characters with a code point greater than 255.

Replies are listed 'Best First'.
Re^4: Perl strings questions
by bliako (Abbot) on Jun 02, 2021 at 17:32 UTC

    Re: python does not mix binary and string, well I thought this hashlib.sha256(encoded).digest() was a binary hash. I am positive that I printed it to see, but right now I have no time, so I will update tomorrow. The rest is very useful and I will read it tomorrow.

    UPDATE: the digest() prints out b'+\xbd\xd0Z\xda;\x05\xbb\x80\x058(.' etc