in reply to Reading and writing to unicode files

The important question is how the MailSweeper files are encoded. "Unicode" is not an answer to that question because there are multiple, completely different encodings for Unicode characters to bytes. Perl uses UTF-8 internally for Unicode strings. UTF-16 stores Unicode characters as 2-bytes. Windows will use UTF-16 for text files.

If you are getting an extra byte between ASCII characters, then the file is most likely using UTF-16. Try this: 

open (CONFIGREAD, "<:encoding(UTF-16)", $fileaddrlist)

[download]

Replies are listed 'Best First'.
Re: Re: Reading and writing to unicode files
by aquarium (Curate) on Mar 05, 2004 at 02:31 UTC
    hmm....been pondering myself the question...is there a utility to detect which unicode standard a file is? is there such a thing?...how about unix "file" or "type" utilities? is there anything as such in perl to detect the "magic" of the file?
[reply]

      You should be able to do it by looking at the first two bytes to detect UCS2/UTF-16;

      A UCS2 file would normally have a Byte Order Marker (BOM) Which is *either* 0xFF 0xFE or 0xFE 0xFF (depending on the endian-ness of your processor.

      Now you fall into the realm of multibyte. In the parsers I've designed, where there is no other meta data (like an XML prolog, or some encoding information from HTTP) is to firstly attempt to parse as UTF8 (it's dead handy that the conversion is relatively trival) -- if it fails (so it finds an invalid UTF-8 character sequence) I fall back to the current locale.

      * Actually Windows boxen use UCS2, not UTF-16; Strictly speaking, UTF-16 is a superset of this, and the UTF-16 is binary backwardly compatible with UCS2.

      --
      RatArsed
[reply]
        No byte markers, it really looks as if it's just taking two bytes per character, rather than one. SO I tried to break it down to an even more basic level, with the following: 
        open (READ, "<:encoding(UCS2)", "z:/test.txt") or die "unable to open 
+file";

foreach (<READ>) {
  print $_;
}

        [download]
        with test.txt being a copy of the config file I'm trying to open. (BTW, UCS2 fails with quite glorious results, involving much beeping :))
[reply]
[d/l]

In Section Seekers of Perl Wisdom