in reply to Re^3: iso-8859-1 code converter
in thread iso-8859-1 code converter

Okay, how do you know when you're using one or the other?

I change the character encoding in Kate. I actually copy, then change encoding, and then paste over.

6 numerics instead of 3 for that 3-character string definitely means "not utf8" (so presumably euc, based on what you've said);

You are right -- EUC gives 3, and utf8 gives 6. However, the ones EUC gives are not the ones I need. :)

And what do you mean by "the values are actually just HTML representation"?

Sorry, that sounds stupid. They are the decimal form of Unicode, which HTML can read as &#XXXXX;You can type in the above string into this little engine and have it spit out the result.
http://www.pinyin.info/tools/converter/chars2uninumbers.html

Are you really still having a problem with this?

I think I've got it now, but I still don't know how I can get this decimal value in Perl. If I pass a utf8 string to ord(), it gives me a value for each byte, not the double-byte character. I could do something tacky and pass my string in question to the aforementioned website to get the decimal value, but that seems overkill, to say the least.

Replies are listed 'Best First'.
Re^5: iso-8859-1 code converter
by graff (Chancellor) on May 06, 2009 at 15:06 UTC
    I think I've got it now, but I still don't know how I can get this decimal value in Perl. If I pass a utf8 string to ord(), it gives me a value for each byte, not the double-byte character.

    Only if perl doesn't know that the string really is utf8. If the string is coming from a data file, and you open that file without telling perl specifically what type of encoding to use for it, perl reads data from the file as bytes, and you need to decode() the string (using the Encode module) so that perl will see it properly converted/interpreted as utf8 characters, and the scalar variable holding the string will be flagged as such. Note the following:

    # open an euc-jp file: open( $inp, "<:encoding(euc-jp)", "some_file.euc" ) or die "$!"; while (<$inp>) { # data will be decoded on input ... # so $_ will have its utf8 flag "on" and # s///, ord(), etc will use character semantics } # open a utf8 file: open( $inp, "<:utf8", "some_file.utf8" ) or die "$!"; while (<$inp>) { # data will be interpreted as utf8 on input ... # so $_ will have its utf8 flag "on" and... (same as above) }

    If your string is based on some literal value in your perl script file, you have to make sure that the editor you use to compose the source code is saving the script as a utf8 file (and you probably need to put use utf8; in the script, so perl knows how to interpret the literal utf8 characters that it contains).

Re^5: iso-8859-1 code converter
by Anonymous Monk on May 06, 2009 at 09:20 UTC