One plausible explanation could be that even though perl actually garbage collects it, it's not released to the OS to be reclaimed just yet.

Allocating memory from the OS -- especially if it's a fixed size that you're likely to want to reallocate soon -- is a great way to make your program run slowly. I sped up all Parrot function calls a few minutes ago by removing a pair of malloc/free calls.