There is almost never a need to use a map and a grep one after the other; they can be combined into a single loop.
my %file = map { /^[^.].*\.pm$/ ? ($_ => (stat $_)[10]) : () } readdir DIRH;
For each item, if the regex matches, return a two element list (that's the map part), otherwise return an empty list (that's the grep part).
I combined the original poster's two regexes as well, and made the match a little more explicit.