Your analysis while good in theory is off in details. Judging by a comparison with a development Perl, map will always be about twice as slow. Probably because of all of the lists you create, then having to internally do the assignments to the hash.

However the slow-down you wondered at with 500 elements is the effect of the quadratic slowdown in my note. The problem is that the entire argument stack was getting moved for each call of the block. The first time that takes work n. The second n-1. The third n-2. etc. So it is quadratic.

If you are willing to recompile Perl, there is a very simple patch available.