My understanding of Perl's filesystem rules are that the programmer is responsible for performing encoding on every string prior to passing it to the OS

Yes, but open can transform that properly-encoded text into garbage. That's a bug.

If you skip that encoding step

I didn't say anything about skipping the encoding step. This has nothing to do with anything I said.

I'll continue anyway, but it's all a straw man.

It isn't a "bug" in the C language

Of course not, because the C language doesn't define the behaviour of this (which is well understood as allowing it to have any behaviour).

However, Perl does define the behaviour of open. It should create a file with the provided name. Provide a properly-encoded string consisting of bytes C3 A9 2E 74 78 74, and it should create a file with the name consisting of the bytes C3 A9 2E 74 78 74. It doesn't always do that, and that's a bug.

Your example is a false parallel.