Re^4: substr function

Replies are listed 'Best First'.
Re^5: substr function by Jim (Curate) on Jan 13, 2011 at 17:11 UTC
Huh? If `<194>` is a character entity that represents a "non-ASCII character," then this is precisely something that makes my question germane to the problem of measuring the length of the text in which the character entity occurs. For Unicode text, there are at least three valid, meaningful ways to measure the size of the text: in bytes, in characters (code points), and in grapheme clusters.	[reply] [d/l]
Re^6: substr function by ikegami (Patriarch) on Jan 13, 2011 at 17:30 UTC
Oops, it's makes bytes vs encoded characters moot, but characters vs graphemes is still relevant. By the way, there is indeed a fourth: Some characters are double-wide, so you could also talk about visual width.	[reply]
Re^7: substr function by Jim (Curate) on Jan 13, 2011 at 17:48 UTC
Exactly! Which is why I was careful to write "there are at least three valid, meaningful ways to measure the size of [Unicode] text." See Unicode Standard Annex #11: EAST ASIAN WIDTH and Unicode Standard Annex #14: Unicode Line Breaking Algorithm for the gory details. Use Unicode::LineBreak and Unicode::GCString in your Perl programs.	[reply]