in reply to Re^2: iso-8859-1 code converter
in thread iso-8859-1 code converter

So you have a string like this: "任天堂", and you have a template for submitting a query containing that sort of string, which is:
http://www.play-asia.com/paOS-19-71-6-49-en-15-{your_string_here}-43-6 +.html
(I hope I'm parsing that correctly, but I have to say it looks a bit implausible.)

And in order to plug that sort of string into that query template, you have to:

  1. break the string into separate characters;
  2. convert each character into its decimal numeric Unicode code point value
  3. embed each numeric character value between "&#" and ";"
  4. do the "uri encoding" that turns "&", "#" and ";" into "%26", "%23" and "%3B", respectively

So, where's the problem? Have you tried something like this?

my $url = http://www.play-asia.com/paOS-19-71-6-49-en-15-XXX-43-6.html my $string = "..."; # wherever you get your Unicode Asian string from $string =~ s/([^[:ascii:]])/"%26%23".ord($1)."%3B"/eg; $url =~ s/XXX/$string/;

Replies are listed 'Best First'.
Re^4: iso-8859-1 code converter
by GaijinPunch (Pilgrim) on May 06, 2009 at 01:12 UTC
    >>rovf
    "Or did I misundertand you here completely?"

    Yes. I will get that from the page listed. I need the ASCII representation to do so though, and that's what I'm after. >>graff
    That seems to be on the right track. Never knew what ord() did. However, as before, I'm using utf8 or euc encoded strings, so ord returns the numeric representation of each byte, not each character. The above gives 6 distinct codes, not 3. My assumption is if I passed it iso-8859-1 encoded strings it might work, but my text editor (Kate on Linux) says they're not valid for that encoding, and chokes.

    Wait, the values are actually just HTML representation (like © is the copyright sign). I guess I should just either plug my values into a calculator like below, or search for the table somwhere.
    http://www.pinyin.info/tools/converter/chars2uninumbers.html

    Sorry my confusion led to such a lengthy discussion. :?
      ... as before, I'm using utf8 or euc encoded strings...

      Okay, how do you know when you're using one or the other? Does the web site (play-asia.com) support both encodings, and if so, how do you tell it which one you're using?

      ord returns the numeric representation of each byte, not each character. The above gives 6 distinct codes, not 3.

      6 numerics instead of 3 for that 3-character string definitely means "not utf8" (so presumably euc, based on what you've said); if the web site is looking for utf8-encoded numerics, you should use Encode to convert from euc to utf8, then use ord():

      use Encode; ... my $string = "..."; # wherever you get your euc string from; $string = decode( "euc-jp", $string ); # now it's a utf8 string; ... # plug it into the url as described earlier

      What does "rovf" mean, and am I missing your point(s) completely? (sorry-- I just realized that "rovf" was another monk -- moving right along...)

      Your descriptions and replies are a bit hard for me to follow. And what do you mean by "the values are actually just HTML representation"? There are numeric character entity references (both decimal and hex, used in HTML, XML, SGML), there are symbolic entity references (like "á"), and there are uri-encoded versions of these (with the punctuation marks converted to "%" followed by two hex digits).

      Are you really still having a problem with this?

Re^4: iso-8859-1 code converter
by GaijinPunch (Pilgrim) on May 06, 2009 at 08:26 UTC
    Okay, how do you know when you're using one or the other?

    I change the character encoding in Kate. I actually copy, then change encoding, and then paste over.

    6 numerics instead of 3 for that 3-character string definitely means "not utf8" (so presumably euc, based on what you've said);

    You are right -- EUC gives 3, and utf8 gives 6. However, the ones EUC gives are not the ones I need. :)

    And what do you mean by "the values are actually just HTML representation"?

    Sorry, that sounds stupid. They are the decimal form of Unicode, which HTML can read as &#XXXXX;You can type in the above string into this little engine and have it spit out the result.
    http://www.pinyin.info/tools/converter/chars2uninumbers.html

    Are you really still having a problem with this?

    I think I've got it now, but I still don't know how I can get this decimal value in Perl. If I pass a utf8 string to ord(), it gives me a value for each byte, not the double-byte character. I could do something tacky and pass my string in question to the aforementioned website to get the decimal value, but that seems overkill, to say the least.

      I think I've got it now, but I still don't know how I can get this decimal value in Perl. If I pass a utf8 string to ord(), it gives me a value for each byte, not the double-byte character.

      Only if perl doesn't know that the string really is utf8. If the string is coming from a data file, and you open that file without telling perl specifically what type of encoding to use for it, perl reads data from the file as bytes, and you need to decode() the string (using the Encode module) so that perl will see it properly converted/interpreted as utf8 characters, and the scalar variable holding the string will be flagged as such. Note the following:

      # open an euc-jp file: open( $inp, "<:encoding(euc-jp)", "some_file.euc" ) or die "$!"; while (<$inp>) { # data will be decoded on input ... # so $_ will have its utf8 flag "on" and # s///, ord(), etc will use character semantics } # open a utf8 file: open( $inp, "<:utf8", "some_file.utf8" ) or die "$!"; while (<$inp>) { # data will be interpreted as utf8 on input ... # so $_ will have its utf8 flag "on" and... (same as above) }

      If your string is based on some literal value in your perl script file, you have to make sure that the editor you use to compose the source code is saving the script as a utf8 file (and you probably need to put use utf8; in the script, so perl knows how to interpret the literal utf8 characters that it contains).

Re^4: iso-8859-1 code converter
by GaijinPunch (Pilgrim) on May 07, 2009 at 00:40 UTC
    graff

    That worked. Thanks!
    Definitely some good nuggets in this thread which went on a bit longer than I had anticipated.

    Thanks to everyone for their suggestions and patience.