So, let's get this straight.

You fixed the way you read and write data from and to files, and now the old data which were written in the broken way don't work anymore.

So you need three steps:

  1. Identify which data was written in the old, broken way
  2. Find out in what way that data is broken
  3. Fix the data

For identifying which data is broken, I hope you have some timestamps along with your data, or that you write the rows sequentially, so that you can easily see which rows are "old" and which are "new".

For identifying in which way the data is wrongly encoded, I recommend to look at the data with hexdump -C yourfilename - it gives more reliable information than perl (at least if you don't grok how perl does what it does).

If you have some samples of the broken data and how it should look like, you can also try Encode::Repair. If that module doesn't work for you, I'll be happy to improve it, if you provide enough information.

Perl 6 - second systems done right

In reply to Re: Mess with UTF-8, utf8 and raw encoding on live working platform by moritz
in thread Mess with UTF-8, utf8 and raw encoding on live working platform by AlfaProject

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.