uva has asked for the wisdom of the Perl Monks concerning the following question:

dear monks,
i can find whether the particular string is utf8 or not by means of utf8::is_utf8();
but how to detect which encoding is used for the particular file. in order to check if the file is encoded in utf8 or utf16 or someother format.

Edited by planetscape - fixed br tags

Replies are listed 'Best First'.
Re: help needed in file encoding
by zentara (Cardinal) on Mar 20, 2006 at 16:37 UTC
[reply]
Re: help needed in file encoding
by ikegami (Patriarch) on Mar 20, 2006 at 17:54 UTC

    You could check if the first few bytes contains a BOM. 

    if    (substr($text, 0, 4) eq "\x00\x00\xFE\xFF") { utf32be }
elsif (substr($text, 0, 4) eq "\xFF\xFE\x00\x00") { utf32le }
elsif (substr($text, 0, 2) eq "\xFE\xFF"        ) { utf16be }
elsif (substr($text, 0, 2) eq "\xFF\xFE"        ) { utf16le }
elsif (substr($text, 0, 3) eq "\xEF\xBB\xBF"    ) { utf8    }
else {
   No BOM found. Might not be UTF.
   Use another method to guess the
   encoding.
}

    [download]

    It's not very reliable. The protocol which encases your stream/data must really specify the encoding for things to work smoothly.

[reply]
[d/l]
      Consider File::BOM if you need to do something like this.
[reply]
        i tried with utf8::is_utf8($string) ,the $string is got from the big5 encoded file,i got the output as 1(true). how does it happen . Is all internal representation is utf8 in windows?.
[reply]
Re: help needed in file encoding
by timos (Beadle) on Mar 20, 2006 at 16:45 UTC
    In Linux: wc -m tells you the number of characters, wc -c the numer of bytes. If the last one is greater then the first one it is probably UTF-8/16 encoded.
[reply]
[reply]
[reply]

Back to Seekers of Perl Wisdom