The issue seems to be specific to the man page formatter (perldoc -o text ... for example (i.e. Pod::Text) appears to work fine for me).  From Pod::Man:

utf8

By default, Pod::Man produces the most conservative possible *roff output to try to ensure that it will work with as many different *roff implementations as possible. Many *roff implementations cannot handle non-ASCII characters, so this means all non-ASCII characters are converted either to a *roff escape sequence that tries to create a properly accented character (at least for troff output) or to X.

If this option is set, Pod::Man will instead output UTF-8. If your *roff implementation can handle it, this is the best output format to use and avoids corruption of documents containing non-ASCII characters. However, be warned that *roff source with literal UTF-8 characters is not supported by many implementations and may even result in segfaults and other bad behavior.

So I guess the question becomes how to tell perldoc to pass the utf8 option to the Pod::Man constructor...