Re^4: Mail::Sender character set/encoding

So, to "just know" is it a matter of being familiar with different character encoding sets and the context of the document in question (for example, a text file created with vi on my machine, an email from China, or a Word attachment from Zimbabwe)?

I took at look at this page http://en.wikipedia.org/wiki/Character_encoding most of that is fairly familiar to me. It did add a little to the confusion that latin1 or us-ascii wasn't mentioned on that page (ok, I know Wiki is not the definitive source).

I suppose like anything else, experience is valuable. So after this, I guess I know a little more.

Comment on Re^4: Mail::Sender character set/encoding

Replies are listed 'Best First'.
Re^5: Mail::Sender character set/encoding by almut (Canon) on Mar 12, 2009 at 21:28 UTC
Ideally, there is meta info, either explicitly (as in Content-Type, etc.), or implicitly (such as where the data originated from, as you're saying). For Unicode encodings, there's also the BOM. In case there isn't, we humans (as opposed to computers) typically excel at figuring out what encoding is being used (at least if we understand the language the text is in) — mainly because our mind has abilities and resources (like world knowledge) that AI is still struggling with... As we can usually tell that only certain characters make sense in certain positions, we can check their byte values against what's documented, and arrive at a decision (or at least an educated guess) rather soon.	[reply]

Replies are listed 'Best First'.

Re^5: Mail::Sender character set/encoding
by almut (Canon) on Mar 12, 2009 at 21:28 UTC

Ideally, there is meta info, either explicitly (as in Content-Type, etc.), or implicitly (such as where the data originated from, as you're saying). For Unicode encodings, there's also the BOM.

In case there isn't, we humans (as opposed to computers) typically excel at figuring out what encoding is being used (at least if we understand the language the text is in) — mainly because our mind has abilities and resources (like world knowledge) that AI is still struggling with... As we can usually tell that only certain characters make sense in certain positions, we can check their byte values against what's documented, and arrive at a decision (or at least an educated guess) rather soon.

[reply]