SOLVED!

Here is the short verson:
MXNet up to version 1.9.1 was written for cuda 11. It is not compatible with cuda 12
MXNet 2.0.0 with cuda12 compatability was retired before it was stable

There was a Docker container on the HUB that ran mxnet, but it's gone now
There is a Docker container that includes the cuda 11 and cudnn8 development 'stuff'

For your convenience*, here is a Dockerfile that will** build an image with working AI::MXNet 1.9.0 (1.9.1 doesn't work)
(And some tips for those who don't use docker. I'm far from expert, but what I've put here is at least correct at the time of writing)
Build with docker buildx build -t perlmx .*³ The final image is 18GB and can take hours to build :-(
Run with docker run --runtime=nvidia --gpus=all -it perlmx*⁴ to get a root prompt. You can then also ssh -X mxnet@172.17.0.2*⁵*⁶*⁷ to get X11
Or run with docker run --runtime=nvidia --gpus=all -dit perlmx to detach and then ssh -X mxnet@172.17.0.2*⁷ *⁸
(If you are new to Docker, read up on the run, start, stop, ps -a and images commands, or you will loose your work, suck up all your disks or both)

You could also use the information and magic versions below to build a bare-metal (or other VM) server if you want. I'm not going to tell you what to do.

Good Luck

# syntax=docker/dockerfile:1
# vim: filetype=dockerfile
# -*- mode: dockerfile -*-
#
# dockerfile to build working perl/mxnet with GPU support


FROM nvidia/cuda:11.1.1-cudnn8-devel AS build

WORKDIR /usr/src/mxnet

RUN <<INSTALL_CPP
apt update

export DEBIAN_FRONTEND=noninteractive
export TZ="Etc/UTC"
apt install -y build-essential git libatlas-base-dev libopencv-dev pyt
+hon3-opencv \
    libcurl4-openssl-dev libgtest-dev cmake swig



cd /usr/src/gtest
cmake CMakeLists.txt
make
cp lib/*.a /usr/lib

INSTALL_CPP


RUN <<BUILD_MXNET
cd /usr/src/mxnet
export BUILD_OPTS="USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=
+1"
export LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/
+usr/local/cuda:/usr/local/cuda-11.1/compat

git clone --branch 1.9.0 --recursive https://github.com/dmlc/mxnet .

#The Makefile isn't tuned for parallel builds, but running -j1 is _ver
+ry_ slow
#In tests, -j8 seems to work anyway! But if it does fail, this will fi
+nish
#the build anyway.
#If you still aren't getting a good build, use
#make -j1 $BUILD_OPTS
#and just wait it out.
make -j8 $BUILD_OPTS || make -j2 $BUILD_OPTS || make -j1 $BUILD_OPTS

BUILD_MXNET

RUN <<PERL_MX
export LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/
+usr/local/cuda:/usr/local/cuda-11.1/compat

cd /usr/src/mxnet/perl-package/AI-MXNetCAPI/
   perl Makefile.PL
   make install
cd /usr/src/mxnet/perl-package/AI-NNVMCAPI/
   perl Makefile.PL
   make install
cd /usr/src/mxnet/perl-package/AI-MXNet/
   perl Makefile.PL
   make install

PERL_MX

FROM nvidia/cuda:11.1.1-cudnn8-devel

RUN <<PREP
echo export LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/li
+b64:/usr/local/cuda:/usr/local/mxnet/lib >> /etc/profile.d/99-cuda-li
+brary-path.sh
echo export XDG_RUNTIME_DIR=/tmp/xdg_runtime_\`whoami\` >> /etc/profil
+e.d/99-gnuplot-xdg-path.sh
echo mkdir \$XDG_RUNTIME_DIR 2\>/dev/null >> /etc/profile.d/99-gnuplot
+-xdg-path.sh

export DEBIAN_FRONTEND=noninteractive
apt update
apt install -y \
        wget unzip sudo \
        openssh-server x11-apps net-tools vim \
        libmouse-perl pdl cpanminus libgraphviz-perl libpdl-graphics-g
+nuplot-perl \
        libpdl-ccs-perl libfunction-parameters-perl \
        libperlio-gzip-perl libgtk2-perl
apt-get clean

cpanm -q Hash::Ordered
rm -rf /root/.cpanm

sed -i "s/^.*X11Forwarding.*$/X11Forwarding yes/" /etc/ssh/sshd_config
sed -i "s/^.*X11UseLocalhost.*$/X11UseLocalhost no/" /etc/ssh/sshd_con
+fig
grep "^X11UseLocalhost" /etc/ssh/sshd_config || echo "X11UseLocalhost 
+no" >> /etc/ssh/sshd_config

useradd -m -s /bin/bash mxnet
echo "mxnet:mxnet" | chpasswd
touch /home/mxnet/.Xauthority
chown mxnet:mxnet /home/mxnet/.Xauthority
chmod 600 /home/mxnet/.Xauthority
adduser mxnet sudo
PREP

RUN --mount=from=build,src=/,dst=/mnt <<COPYING
mkdir -p /usr/local/mxnet/lib/
cp -p \
        /mnt/usr/src/mxnet/perl-package/AI-NNVMCAPI/blib/arch/auto/AI/
+NNVMCAPI/NNVMCAPI.so      \
        /mnt/usr/src/mxnet/perl-package/AI-MXNetCAPI/blib/arch/auto/AI
+/MXNetCAPI/MXNetCAPI.so   \
        /mnt/usr/src/mxnet/lib/libmxnet.so                      \
        /mnt/usr/src/mxnet/build/libpass_lib.so                 \
        /mnt/usr/src/mxnet/build/libcustomop_lib.so             \
        /mnt/usr/src/mxnet/build/libcustomop_gpu_lib.so         \
        /mnt/usr/src/mxnet/build/libsubgraph_lib.so             \
        /mnt/usr/src/mxnet/build/libtransposerowsp_lib.so       \
        /mnt/usr/src/mxnet/build/libtransposecsr_lib.so         \
                /usr/local/mxnet/lib/

cp -rp /mnt/etc/alternatives/lib*               /etc/alternatives/
cp -rp /mnt/usr/lib/lib*                        /usr/lib/
cp -rp /mnt/usr/lib/x86_64-linux-gnu            /usr/lib/
cp -rp /mnt/usr/local/lib/x86_64-linux-gnu      /usr/local/lib/
cp -rp /mnt/usr/local/share/man                 /usr/local/share/
cp -rp /mnt/usr/local/share/perl                /usr/local/share/

cat <<"STARTUP" > /root/startup.sh
#!/bin/bash
echo -e "\n==== ===== =======\nPerl/MXNet startup\n==== ===== =======\
+n"
ln -s /dev/null /dev/raw1394 2>/dev/null
mkdir /tmp/runtime-mxnet 2>/dev/null
cat /etc/*elease
echo
ifconfig -a | perl -ne 'if (/^\s+inet\s+(\d\S+\d)\s/) { print "IP: $1\
+n"; }'
echo
service ssh start
echo
bash $*
STARTUP

chmod 755 /root/startup.sh

COPYING

EXPOSE 22/tcp

WORKDIR /root

ENTRYPOINT ["/root/startup.sh"]
[download]

* Enjoy. Also, I added PDL and PDL::Graphics::Gnuplot
** You need to install the NVIDIA Container Toolkit if you want to actually use your GPU.
*³ If you are new to docker: Just install docker (v23 or later) and it should just work
*⁴ If you don't want to use your GPU, don't have a compatible GPU or couldn't get the NVIDIA container toolkit to work, run with docker run -it perlmx or docker run -dit perlmx and make your CPU do all the things.
*⁵ Password is mxnet.
*⁶ Your IP may vary.
         When you first launch the container interactively (with -it), it displays its IP address.
         You can also use docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' container_name_or_id on a running container
*⁷ You can also not use -X if you don't want/need X11 support.
*⁸ User mxnet is in sudoers, so...

FWIW: I'm running the docker service in kali linux in WSL2, with an nVidia 2080 Super. YMMV