No explanation (yet),

You end up at one of these:

PerlIO *
PerlIO_openn(pTHX_ const char *layers, const char *mode, int fd,
             int imode, int perm, PerlIO *old, int narg, SV **args)
{
    if (narg) {
        if (narg > 1) {
            Perl_croak(aTHX_ "More than one argument to open");
        }
        if (*args == &PL_sv_undef)
            return PerlIO_tmpfile();
...
[download]

PerlIO *
PerlIO_openn(pTHX_ const char *layers, const char *mode, int fd,
             int imode, int perm, PerlIO *f, int narg, SV **args)
{
    dVAR;
    if (!f && narg == 1 && *args == &PL_sv_undef) {
        if ((f = PerlIO_tmpfile())) {
[download]

Notice how they check if the file name SV is the global constant undef (*args == &PL_sv_undef) rather than checking if it's undefined (SvOK(*args)).

It could be a bug, but I suspect it's intentional to try to detect the case where the file name is accidentally undef. It almost never makes no sense for a program to allow both a valid file name and undef. If your program is one of those, you can use

open(my $fh, '+>', $fn // undef)
[download]

I think they should used a special character in the mode string instead of using a literal undef.

but it doesn't say anything about what happens if the undef isn't written literally...)

True, but one doesn't expect a difference in behaviour between the undefined value returned by undef and an undefined scalar. Any differences should be explicitly stated.