There's a bug in some versions of glibc's memory allocation system that makes freeing lots of small data pieces take an extraordinarily long time. (2.2.5 looks to be one of the prime culprits) There's not a whole lot perl itself can do about this. What you can do is either upgrade your libc (newer versions have a new memory allocation system which seems to fix this) or build perl with perl's own malloc.

What's happening is that perl's allocated lots of little chunks of memory. When perl exits, it needs to go through and hand them all back to the system like a well-behaved potentially embedded program should. Unfortunately the afflicted versions of libc, when presented with a zillion frees, seem to go insane managing its free list, with the performance problems you've noted. (This can also happen if you free up a lot of little chunks of memory in the middle of your program, in which case you'll get a noticeable pause)