in reply to Re: Perl Modules for handling Non English text
in thread Perl Modules for handling Non English text

That's not enough for all languages and hence "wide characters", or 16 bit ones.

Perl's wide chars are 32-bit or 64-bit depending on the build, not 16. 

fmdev10$ perl -le'print ord "\x{FFFFFFFF}"'
4294967295

[download]
persephone$ perl -le'print ord "\x{FFFFFFFFFFFFFFFF}"'
18446744073709551615

[download]

Unicode currently requires 17 bits.

is each byte a character or is two bytes a character?

Or something else entirely, as in the following popular encodings: UTF-8 (1-4 bytes per char currently, 1-6 possible), UTF-16le/UTF-16be (2 or 4 bytes per char).

Replies are listed 'Best First'.
Re^3: Perl Modules for handling Non English text
by Marshall (Canon) on Mar 31, 2009 at 01:51 UTC
    My answer was based upon ANSI C:
    GCC on my Intel machine:
    #include <stdio.h>
#include <stddef.h>
int main ()
{
   printf ("hello world\n");
   printf ("size of a wide char is %d bytes", sizeof(wchar_t) );
   return (0);
}
/* prints: 
 hello world
 size of a wide char is 2 bytes
*/

    [download]
    $perl -le 'print ord "\x{FFFFFFFF}"'
    4294967295
    is just a 32 bit unsigned hex number.

    I don't know how many bits Hindi requires.

    Update: http://ascii-table.com/unicode.php shows unicode standards. This is complex. But basically 16 bits does it.

[reply]
[d/l]

      My answer was based upon ANSI C:

      In this Perl discussion, that's as relevant as Java using 32-bit wide chars. I can understanding the mistake of bringing it up initially, but why bring it up again.

      And it's wrong. ANSI C says nothing about wchar_t being 16-bit. sizeof(wchar_t) can be as small as 1, and it's commonly 4. In fact, your own program betrays you. Also from gcc on an Intel: 

      $ gcc -o a a.c

$ a
hello world
size of a wide char is 4 bytes

      [download]

      4294967295 is just a 32 bit unsigned hex number.

      And how did I get that number? By getting the character number of "\x{FFFFFFFF}". Therefore, I had a 32-bit character.

      But basically 16 bits does it.

      Your own reference contradicts you. 17 planes of 16 bits = way more than 16 bits. (21, to be precise.)

      For example, these Chinese chars require more than 16 bits.

      I don't know how many bits Hindi requires.

      It varies by encoding, and it can even vary withing an encoding. But it's completely irrelevant. Perl supports all Unicode characters, including the Hindi ones.

[reply]
[d/l]
[select]
        I think the most commonly used standards are relevant. Most languages can be represented in 8 bits and the vast majority of those that can't be (and Hindi is one of them) can be represented in 16 bits. Yes, I agree that a full representation requires 4 bytes and that Perl can do it. That is not in question!

        At the "end of the day", I normally work with databases generated by other software that can't do 32 bit characters. Maybe you don't have that limitation, but I do.

        The original question was how to handle Hindi and the answer is that Perl does fine and "C" does fine with that as this only requires 16 bits.

[reply]

In Section Seekers of Perl Wisdom