Re^5: create clone script for utf8 encoding

I tried pre tags in the writeup but must not have pasted it in and previewed correctly.

PerlMonks engine automatically replaces all symbols not representable in ASCII by their HTML entity codes: ы → ы. The <code> are special non-HTML tags that don't allow HTML entities inside them to be interpreted, but the transformation still takes place. (How did I write that? <tt>ы</tt> &rightarrow; <c>ы</c> and let PerlMonks make the replacement, knowing that the entity code inside <tt>...</tt> will be interpreted back into ы, while the one inside <c>...</c> won't. How did I write what I just wrote? Lots of <s and <code> = <c> equivalence.)

It is the unicode replacement character: U+FFFD.

The replacement character is what happens when your terminal emulator tries to decode KOI8-R-encoded bytes as UTF-8 and fails. The actual output of diff contains both KOI8-R- and UTF-8- encoded bytes and can be decoded as KOI8-R:

$ diff 1.file 3.file | iconv -f koi8-r
1c1
< Привет
---
> п÷я─п╦п╡п╣я┌

Comment on Re^5: create clone script for utf8 encoding Select or Download Code

Replies are listed 'Best First'.
Re^6: create clone script for utf8 encoding by ikegami (Patriarch) on Dec 20, 2018 at 17:34 UTC
PerlMonks engine automatically replaces all symbols not representable in ASCII by their HTML entity codes Nit: It's actually the browser doing that, and it's for characters outside of cp1252 (not ASCII). PerlMonks pages are returned as cp1252, so the browser knows that forms must be submitted using cp1252. Characters outside of cp1252 (e.g. "ы") can't be submitted, but rather than throwing an error if such characters are provided, the browser provides an HTML encoding of the character (e.g. "ы") instead just in case that works. Of course, it doesn't work within code tags because PerlMonks escapes "&" in code tags so that it shows up as "&".	[reply]

Replies are listed 'Best First'.

Re^6: create clone script for utf8 encoding
by ikegami (Patriarch) on Dec 20, 2018 at 17:34 UTC

PerlMonks engine automatically replaces all symbols not representable in ASCII by their HTML entity codes

Nit: It's actually the browser doing that, and it's for characters outside of cp1252 (not ASCII).

PerlMonks pages are returned as cp1252, so the browser knows that forms must be submitted using cp1252. Characters outside of cp1252 (e.g. "ы") can't be submitted, but rather than throwing an error if such characters are provided, the browser provides an HTML encoding of the character (e.g. "ы") instead just in case that works. Of course, it doesn't work within code tags because PerlMonks escapes "&" in code tags so that it shows up as "&".

[reply]