in reply to Guess between UTF8 and Latin1/ISO-8859-1

A couple of pointers: Personally and professionally, I take the stance that any XML file that doesn't start with the endianness 2-byte code is NOT unicode. Anything within that group that doesn't say something about it's encoding in the text declaration will be interpreted as being 7-bit ASCII, and any character entities I encounter that exceed the 7-bit range (like Í or whatever) are invalid and the whole file is rejected unless clear agreements have been made about the actual encoding of the content.

If you make any other assumptions you will be miserable later. I know I have :-/

The numeric entities can open up a can of worms that's hard to close after the fact: you can decide on an ASCII encoded XML file, but the actual content can be unicode, LATIN-1, japanese or whatever, so you need to decide on the encoding of the content seperately. (Please someone, correct me if I'm wrong. This has been bugging me for too long)

Just my €0.02.
Joost.

"What should it profit a man, if he should win a flame war, yet lose his cool?"
  • Comment on Re: Guess between UTF8 and Latin1/ISO-8859-1

Replies are listed 'Best First'.
Re: Re: Guess between UTF8 and Latin1/ISO-8859-1
by iburrell (Chaplain) on Jan 21, 2004 at 23:04 UTC
    The byte order mark is only used for the UTF-16 encoding, the two-byte Unicode encoding. UTF-8 is the default encoding if the byte order mark or encoding parameter is not present. You are correct that if the encoding is not specified, and the file is not valid UTF-8, then it is an error.

    Numeric entities are always Unicode characters. Unicode is the only character set used in XML. Different encodings can be specified, but they should be mapped into Unicode so the parser deals with the Unicode.

[reply]
Re: Re: Guess between UTF8 and Latin1/ISO-8859-1
by Jenda (Abbot) on Jan 21, 2004 at 21:26 UTC
    Thanks for your comments. Anyway ...
    1. Sorry. We are not gettting any 2bytes and I don't think we are ever going to.
    2. I dont need to care about those at the moment. All those companies are US based. And I think it's quite likely that if some company uses something else that Latin1 they will know how to specify that in the XML properly. Or that they will use some accentuated characters in the test jobs so we do find out that something is wrong during testing.
    3. The whole point of this is that I do not want to reject stuff I don't have to. The imported stuff will be displayed to the users for review and modification so if we do screw up badly they will notice, but we want to do our best to get the data inside. Of course the scripts will notify me if they have to tweak the XML and I will push the other companies to fix their stuff, but the users should not notice anything if possible.
    4. Again in this case it's safe enough to assume that if the company knows what the heck are entities they will be able to specify the encoding properly.

    Jenda
    Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
       -- Rick Osborne

    Edit by castaway: Closed small tag in signature

[reply]
[reply]

        For some applications your position "I can't have any incorrect data in the system, if it aint 100% sure I'll reject it" is definitely the right one. All I'm trying to say is that for me in this particular case to be safe means to import everything I possibly can even if it means that there might be an incorrect character now and then.

        There is not one correct way to handle errors. The right way depends on the application.

        Jenda
        Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
           -- Rick Osborne

        Edit by castaway: Closed small tag in signature

[reply]

In Section Seekers of Perl Wisdom