Perl doesn't really have a garbage collector. It uses reference counting and frees memory when nothing has a reference to it. This does work better with swap because there is no process that scans all the memory looking for unused memory to free.

Perl does not return memory to the OS once it has been freed. It does do a good job of reusing memory, so it is possible to have pages swapped out with only free space or unused data.