Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?

Re: How can I create an UTF8 encoded txt file contains strings like "aaaa"?

by graff (Chancellor)
on Sep 05, 2007 at 03:57 UTC ( #637055=note: print w/replies, xml ) Need Help??

in reply to How can I create an UTF8 encoded txt file contains strings like "aaaa"?

If "visual studio" (presumably a Micro$oft product) happens to be "formally dependent" on using a "Micro$oft-Sanctioned" notion of utf8-encoded input file format, then you may need to ensure that the file begins with a "byte-order-mark" (BOM) character (U+FEFF) -- see whether this helps:
open( OUT, ">:utf8", "a.txt" ) or die "a.out: $!"; print OUT "\x{feff}aaaa\n"; close OUT;
For some reason, M$ apps seem to have adopted the use of a file-initial BOM character to signal that a "plain-text" data file contains utf8-encoded unicode characters. If a file contains utf8 wide characters without the initial BOM, apps like wordpad, etc, will misinterpret the wide characters as something else. And maybe "visual studio" is insisting that a file be "marked as containing utf8" even when it doen't need to include wide characters...

(Of course, the BOM was originally intended to be of use only in UTF16-encoded unicode data files, to indicate the "endian-ness" (byte-order) of the 16-bit data, and it shouldn't really be needed at all in a utf8-encoded file, because utf8 is not affected by big-endian vs. little-endian byte-order. But a number of applications -- particularly M$ apps that are able to handle plain-text files along with their rogues-gallery of "application-specific file formats" -- have inexplicably come to depend on a utf8-encoded BOM at the start of the file, acting like a sort of "magic number" to let them know that they are looking at a utf8-encoded file.)

  • Comment on Re: How can I create an UTF8 encoded txt file contains strings like "aaaa"?
  • Download Code

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://637055]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (4)
As of 2023-03-23 20:20 GMT
Find Nodes?
    Voting Booth?
    Which type of climate do you prefer to live in?

    Results (60 votes). Check out past polls.