This node by Nik confused me greatly.

Apparently, if you post unicode characters in a <code> block, you see the numeric HTML entities instead of the characters. I expect this is due to some kind of double encoding bug.

Example:

Here are some random unicode characters I randomly selected:

حيض,πβΫ

Here are the same characters in a <pre> block: 

حيض,πβΫ

And here are the same characters in a <code> block: 

&#1581;&#1610;&#1590;,&#960;&#946;&#939;

[download]

Note that I didn't use any HTML numeric references myself - I just copy/paste the characters from the gnome "charmap" program into the textfield.

Below is the relevant part of the HTML source to this page: 

<p>
Here are some random unicode characters I randomly selected:
<p>
&#1581;&#1610;&#1590;,&#960;&#946;&#939;
</p>
Here are the same characters in a <tt class='inlinecode'>&lt;pre&gt;</
+tt> block:

<p>
<pre>&#1581;&#1610;&#1590;,&#960;&#946;&#939;</pre>
<p>
And here are the same characters in a <tt class='inlinecode'>&lt;code&
+gt;</tt> block:
<p>
<pre class="code"><div class='codeblock'><tt class='codetext'>&amp;#15
+81;&amp;#1610;&amp;#1590;,&amp;#960;&amp;#946;&amp;#939;
</tt></div></pre>

[download]

As you can see, in the <code> block, the numeric entitie's & chars are incorrectly escaped. I think is is a pretty serious issue now that perl can handle native utf8 source.

Also, I notice that these characters are also doubly escaped in the textarea field that I'm typing in now (i.e. you can enter unicode chars in the textfield, but at preview, in the textfield, you'll just see a bunch of &#number; entries).

"What should it profit a man, if he should win a flame war, yet lose his cool?"

Replies are listed 'Best First'.
Re: Unicode characters in <code> blocks (browser)
by tye (Sage) on Oct 08, 2006 at 05:22 UTC

    It is not a double-encoding bug. Your browser makes the (valid, in this case) assumption that the site expects input in Latin-1 (since that is what it declares its output to be in) and so your browser encodes things that don't fit into Latin-1 as HTML entities (making another assumption that is only partially valid here; that what you are sending will be interpretted as HTML). (Note that not all browsers make these assumptions and so not all browsers will behave this way; though other choices of browser behavior usually have worse outcomes.)

    What you send for node content is interpretted as something close to HTML, but the content of CODE tags is just data; it is not interpretted at all beyond looking for the matching ending CODE (or C) tag. And PerlMonks content is Latin-1 so you can't put anything but Latin-1 into PerlMonks CODE tags.

    We could try to convert the site to UTF-8, but that's a major project and given how long minor projects take I wouldn't start holding my breath on that. We could come up with a scheme for embedding non-Latin-1 characters in CODE tags (or something like CODE tags except for the Latin-1-ness). Most ideas I've come across for that are ugly in one way or another, usually looking less desirable than converting the site to UTF-8.

    - tye        

[reply]
      (Note that not all browsers make these assumptions and so not all browsers will behave this way; though other choices of browser behavior usually have worse outcomes.)

      Indeed, links-derived browsers behave differently. If you enter non-latin1 characters to a latin1 field, it encodes them as flying accents, not as html entities. (Links already has these tables about converting unprintable characters to approximate ascii sequences becasue that's what it does if a html page contains a character that the encoding of your terminal doesn't support.)

[reply]
[reply]

        I don't think it's the browser behaiviour that complicates things. You can change the behaiviour of the browser with the accept-charset attribute of the FORM tag to sending data as UTF-8, and then perlmonks would convert that data to 8859_1. The problem is how perlmonks represents posts and renders them, which cannot easily be changed.

[reply]

Back to Perl Monks Discussion