Courage has asked for the wisdom of the Perl Monks concerning the following question:

Deal possessors of Perl Wisdom!

Please advice on my hopefully simple following question.

Modern perls have an "UTF8" flag attached for each scalar. perldoc perlunicode clearly states this.
There are functions to convert to/from utf8 in place, and there are functions that just set on or off "UTF8" flag

But how do I distinguish whether or not should I treat a data as Unicode returned from some C library?

As an example, sssume I had few scalars written into database, some of them just binary data, some are Unicode, and after that I get my scalars back from DB. How do I distinguish which is simply binary and which is Unicode?

Should I always store "utf8-ness" information always with my scalars?
Is there exists a better and cleaner way?

Thanks in advance,
Courage, the Cowardly Dog

Replies are listed 'Best First'.
Re: How to recognize Unicode data received from any C library?
by liz (Monsignor) on Aug 24, 2003 at 20:18 UTC
    You might want to consider adding a BOM to your Unicoded characterstreams. But that would only make sense if your application is the only one writing to the fields in the database.

    Otherwise you're stuck with store the utf8-ness of information somewhere else. Although some database (drivers) may "know" that it's UTF-8 as well (when having been fed Perl UTF-8-marked strings through DBI) and may restore that information when reading (taking away your "need" to know this). Other monks may know this for sure.

    Liz

[reply]
      Indeed.

      I thought bit more and realized that I should somehow "ask" C library about Unicode-ness of data: as long as my question arises, underlying "C" library should support it and should know about Unicode-ness of its data, regardless of method (BOM or storing somewhere else).

      DB was just an example of my more general problem.

      Thank you, you really helped me a bit further.

      Courage, the Cowardly Dog

[reply]

Back to Seekers of Perl Wisdom