...I seem to recall that map was just rewritten to check for void
context and avoid the extra work if possible.

    That is correct. And it's in Perl 5.8.1!
    From the 5.8.1. perldelta:

        "map" in void context is no longer expensive. "map" is now
        context aware, and will not construct a list if called in
        void context.