in reply to Mess with UTF-8, utf8 and raw encoding on live working platform

So, let's get this straight.

You fixed the way you read and write data from and to files, and now the old data which were written in the broken way don't work anymore.

So you need three steps:

  1. Identify which data was written in the old, broken way
  2. Find out in what way that data is broken
  3. Fix the data

For identifying which data is broken, I hope you have some timestamps along with your data, or that you write the rows sequentially, so that you can easily see which rows are "old" and which are "new".

For identifying in which way the data is wrongly encoded, I recommend to look at the data with hexdump -C yourfilename - it gives more reliable information than perl (at least if you don't grok how perl does what it does).

If you have some samples of the broken data and how it should look like, you can also try Encode::Repair. If that module doesn't work for you, I'll be happy to improve it, if you provide enough information.

Perl 6 - second systems done right
  • Comment on Re: Mess with UTF-8, utf8 and raw encoding on live working platform
  • Download Code

Replies are listed 'Best First'.
Re^2: Mess with UTF-8, utf8 and raw encoding on live working platform
by AlfaProject (Beadle) on Jun 02, 2011 at 14:03 UTC
    Thanks for the answer, but before this I need to understand when I need to write :encoding(UTF-8) and when not.
    For example I wrote a script that take info from all users and put it in index.
    open FH , "<:encoding(UTF-8)","$dir/private_data";
    Process the data
    open FH ,">:encoding(UTF-8)","$path_index";
    After that another web script take the data from this index file
    open FH ,"<:encoding(UTF-8)","$path_index";
    And prints all the data into HTML page. All the German chars with dots and other unique chars , printed as question mark inside polygon.
    But if I use the regular way
    open FH ,"<","$path_index";
    All the chars shows as it needed to be... So I don't really understand why it happens.
[reply]
[d/l]
[select]
      There are two types of string variables in Perl. One type contains text, the other contains bytes.

      Files contain bytes. If you want to write a text string to a file, the string needs to be converted from text to bytes. That's what the >:encoding(UTF-8) does.

      OTOH if you want to read data from a file, without the :encoding(UTF-8) the string will contain bytes, and with it the string contains text.

      Sad thing is, you can't reliably see from looking at a string if it's text or bytes. And if you mix the two up, you will see some broken output.

      So if you use text strings internally in your program, you need the :encoding(UTF-8) both for reading and writing files, and you need to decode all other byte strings that come into your program (for example with %ENV or @ARGV).

      OTOH some modules already decode strings for you (for example XML and JSON parsers), so you must be aware which module does that.

      Perl 6 - second systems done right
[reply]
[d/l]
[select]
        I found in some Russian article to add this line
        use encoding 'UTF-8'
        in each file ... Now all works
        Thanks for all
[reply]
[d/l]

      I don't think anyone can tell you precisely when you need to add an encoding layer on files open for read because it depends on whether the data in the input file is UTF-8 (in your case) encoded or not. If private_data is UTF-8 encoded then you need to add the UTF-8 encoding layer to decode it on input. If path_index is intended to be UTF-8 encoded file you need to encode the data when writing it.

      When you say "prints all the data into HTML page" are you running some sort of CGI, in web server code because if you are you need to set the the doctype in the HTML and probably on the content-type too.

[reply]

In Section Seekers of Perl Wisdom