liz has asked for the wisdom of the Perl Monks concerning the following question:

A client of mine has a database with a great number of (MySQL) tables which may contain invalid UTF-8 data. Since this database is filled from very many different sources, it is not (yet) feasible to reliably stop invalidly coded data to enter the database.

Since some XML feeds are generated from this data, it is imperative that UTF-8 is obtained from the database. So, a small script was developed that regularly reads all tables and all text fields to check for correct UTF8ness and remove illegal characters when needed.

It would seem that such a script would be handy to have as a module. Would it make sense to generalize this into a module? And if so, what would be a good namespace for it? Or is there such a beast already and have we reinvented a wheel?

Liz

  • Comment on Invalid UTF8 data: namespace suggestion needed or wheel reinvented?

Replies are listed 'Best First'.
Re: Invalid UTF8 data: namespace suggestion needed or wheel reinvented?
by PodMaster (Abbot) on Mar 02, 2004 at 11:00 UTC
    How is the utf-8 usually being corrupted?
    What exactly are you doing (are you just removing illegal characters? what else)?
    Rules (is it configurable, or will it be, how)?

    I don't know about a module name (Convert::FixMalformedUTF8? Encode::FixMalformedUTF8?) cause it sounds like a script to me (cpan likes those), cause you can do all that with the Encode/Unicode and modules like that (that's what you probably did).

    MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
    I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
    ** The third rule of perl club is a statement of fact: pod is sexy.

[reply]
      How is the utf-8 usually being corrupted?

      A good question. It's a system that has it roots in Perl 4 and which consists of hundreds of pages (CGI scripts) that are being used by many different people 24x7. And which is currently being migrated to a more manageable situation, so there is little point in trying to find out which users are doing what on which page causing the problem.

      What exactly are you doing (are you just removing illegal characters? what else)?

      Indeed. They're mostly free text entries of different sources in various languages.

      Rules (is it configurable, or will it be, how)?

      The only configuration that I see, would be the ability to list the tables to correct (defaulting to all). And how fields fixed are reported (either STDERR or calling a subroutine).

      ...it sounds like a script to me

      Well, it is is a script now. But I dislike scripts as such, because you are never flexible enough. I'd rather have a simple script that calls a module, than that people start copying a script and start adapting it to their needs.

      Liz

[reply]
Re: Invalid UTF8 data: namespace suggestion needed or wheel reinvented?
by mirod (Canon) on Mar 02, 2004 at 11:21 UTC

    This seems very specific. If the data is invalid utf8 then the script has to guess what the problem is, which is highly dependent on the process that created the data in the first place.

    Is the data in a different encoding? Does the script use Encode::Guess? What does it do exactly?

    My first reaction would be that such a script could be quite dangerous, giving users the impression that they can just run it and have all their problems disappear, when I believe that cleaning up encodings is actually a task that is very difficult to automate in general. At the very least it should come with a huge warning and a long definition of the exact scope of the cleanup it performs

[reply]
      Does the script use Encode::Guess?

      Good point. I had forgotten about that module. Will look into that.

      What does it do exactly?

      With Encode: $string = decode("utf8", $string,FB_DEFAULT )

      ... cleaning up encodings is actually a task that is very difficult to automate in general.

      I agree. But since XML is very picky about encoding errors, and the XML feed must continue in some way, this is (for now) the solution.

      It's a problem that many people (will be|are) facing when migrating legacy systems to Unicode/XML aware systems, which is why I think it warrants someting on CPAN.

      Liz

[reply]
[d/l]

Back to Seekers of Perl Wisdom