The Perl-XML FAQ has a section on encodings.

When you parse a file, the resulting data in the Perl variables will be UTF8 encoded regardless of the source encoding. I'm not an expert on mySQL but I wouldn't have thought that the act of INSERTing into a table would result in characters being converted from UTF8 to ISO8859-1.

With Perl 5.6.0 and later, you can convert a UTF string to a latin-1 string with the somewhat cryptic:

use utf8;

my $latin = pack("C*", unpack('U*', $utf));
[download]

As jkahn said, it's not possible to map all UTF characters to Latin-1. In particular, the 'smart quotes' characters from MS Office apps do not have Latin-1 equivalents. You could simply encode characters beyond 0x7f as numeric entities (if you're ultimately going to write them back out as XML or HTML) or you could replace troublesome characters with more generic equivalents. The FAQ has some code snippets for both options.

thank you very very very much!

this solved my headache, and now everything is inserted into the database correctly. My task for aftersleep is to dig through the FAQ at perl-xml, and learn more about everything.

You can't imagine how happy I am right now :)

About the additional characters that won't fit into Latin-1, there won't be any occurence of such characters. But still i'm going to read up on this, since it's possible that something like this might occur one day.

Emanuel

A couple of notes, some of which may be relevant. Forgive me if you know all this -- I figured it might be useful to somebody who's looking for this kind of information, even if it doesn't necessarily help Emanuel:

1. ISO-8859-1 doesn't have the richness to encode every possible character from UTF-8. Many Eastern European characters (not to mention South and East Asian characters) cannot be encoded in ISO-8859-1. There just aren't enough bits. Are there characters in your data outside the range (U+0000 .. U+00FF) ?
2. Perl's internals are in UTF-8, so the fact that print outputs correct-looking data may be because Text::Iconv is not doing it's job correctly (or you're not using the feature the way it's intended). In other words, if your data is *still* in UTF-8, then it will probably print correctly.
3. If you're using .nix or cygwin, then you probably have the od (octal dump) tool available, which I find indispensable for determining codeset issues (editors like vi and emacs tend to operate at too high a level, because they try to interpret the encoding for you and things "look fine" even when they're in the wrong encoding). I use:od -a all the time to figure out whether I've used encoding tools correctly.

HTH, jkahn

Hello, thanks for the reply

It has to be ISO-8859-1, or Latin-1. I don't have any Asian Characters in it.

The second point mention could be true I guess. Think i'll have to do more digging into that. Hope i'll find something this way, although I thought I'm doing it the right way, I might easily be wrong.

I've been using hexdump and calculated to octal, didn't know about od.. you never stop learning.

thanks for your reply, i'll get working on Text::Iconv.

Emanuel
Edit:
Here's a sample output, i quickly hacked in..:

Before Conversion: Live Fußball: Bundesliga, 3. Spieltag
After Conversion: Live Fußball: Bundesliga, 3. Spieltag

dumped it to a file, checked it with od and hexdump, but it looks correct.. still in the dB it appears as in Bevore Conversion.

The trickery is all connected to the UTF8 flag, which is a bit flag attached to each string. For the data that XML::Parser returns, that UTF8 flag is set.

To get rid of this behaviour, clear the UTF8 flag. One way you can do it, is like this:

sub de_utf8 {
    use bytes;
    return "$_[0]";
}
[download]

my $decoder = new MIME::Decoder 'quoted-printable' or warn "\nunsuppor
+ted\n";<br>
open (TEMP, ">/tmp/$$") or warn "\ncan't open /tmp/$$ for writing";<br
+>
$decoder->decode(\*MSG, \*TEMP);<br>
[download]