For testing purposes I've put some users in an HoH

That was a good idea. However, you do not say if you have also written use utf8; in your source. Did you do that?

but I still end up with something like "K?hne" in Azure

Try posting it to something which you control rather than Azure. That way you can determine if the problem is internal to Azure or not. Even just saving it to a local file with a known encoding would do for starters.

Unicode looks big and scary the first time you have to deal with it but it's not so bad in the long run. Good luck.

(edited for typo)

When reading up on unicode I came across something I took as a warning not to use "use UTF-8". So I did not. The remark about using utf-8 in the code did confuse me: using Kühne in a string is using utf-8 in the code?

Must give your suggetion to take Azure out of the equating some thought, not really sure how to go about that.

using Kühne in a string is using utf-8 in the code?

It is, if it is written with utf-8 encoding. use utf8; tells perl that your source code includes literal utf-8 characters. If you don't include that but insert literal utf-8 characters in your code, all manner of bad things will ensue. See utf8.

There is almost no reason not to use uft8; in all your code these days.

---

🦛

The most important thing to know about Unicode in perl is that you are responsible for remembering whether a particular scalar contains "bytes", or "Unicode characters". Perl does not track this for you. If you lose track of this, you're going to have a confusing time.

So to start, if you write a unicode character in your perl source file, you have "bytes" (which may contain a valid utf-8 sequence from your code editor, but it's still "bytes"). If you put use utf8; at the start of your file, then perl understand that you are defining all your strings with "unicode characters".

"encode" means that you take Unicode Characters, and convert them to utf-8 byte sequences. i.e. the input is characters, the output is bytes. Decode is the opposite. (you may know that perl stores unicode characters internally as utf-8, but this is an implementation detail and only confuses the issue. Think in terms of characters and bytes.)

When you perform I/O, including "warn" and "print" and "readline", you are reading/writing bytes, unless you put the binmode $fh, ":encoding(UTF-8)" layer on that file handle. If you have a string known to be unicode characters, and you want to write it to a file handle without the special encoding layer, you need to "utf8::encode($str)" it yourself before writing.

When you talk to a database, you should make sure that you know whether you are receiving unicode or bytes for any given field. Most database drivers can be configured to correctly identify text columns (and assume them to be unicode) vs. binary columns of bytes.

When you write to a web API, you need to keep track of whether that API expects characters or bytes. It all has to be bytes before it goes over the wire, but sometimes the API does that step for you.

When in doubt, hex-dump the string to find out whether Perl thinks the character is "\x{100}" or "\xC4\x80". A handy utility for this is B::perlstring, though it outputs bytes in octal rather than hex.

use B "perlstring";

my $x= "\x{100}";
say perlstring $x;

utf8::encode($x);
say perlstring $x;
[download]

Thanks all for the replies, I think it cleared my mind of some confusion.

Just tried to do as you guys suggested: keeping track on bytes or chars, using utf-8 in my source. And voila, things got kinda simple. Told the backend (graph) to expect utf-8 and was just able to send the scalar (containing utf-8 chars). No further encoding or decoding needed

Guess I will have to make sure when I actually download information from the school information system that it is in fact utf-8 chars.

Back to assigning groups and other properties to our students.... thank!

Peter

As others have said, local encoding matters.

But you also have to make sure that the POST request tells the server which character encoding you are using. I think the default is still ISO8859-1. For the downlink portion, the server sets a header like this for UTF8:

Content-Type: text/html; charset=utf-8
[download]

For a post request, you also have to set the encoding in the client. Depending on how the content of your post is structured, you have to adapt this header to your needs:

Content-Type: application/x-www-form-urlencoded ; charset=UTF-8
[download]

One easy way to do this is to look what "Content-Type" header LWP is currently sending, and then adapting it so it explicitly says "UTF-8" for the charset. It is also rather important to make sure that the data you send is, in fact, utf8 encoded.

Not that it really satifies me... but I did get something working... right after asking

Turned out the it does work when I decode using UTF-8 when I create the request. (And removing all the left over decodes and encodes scatttered in my code). Just don't get why this works. Decode seems to me the opposite of what I need.