Two things to know about Perl memory management:

• When memory is freed, it can be released to perl's free memory pool rather than being released to the OS. Future allocations will dip into this pool first.

• Heap fragmentation can occur, which means memory may need to be allocated for a structure even though the total memory available within then heap is big enough to hold the structure.

I don't know if that helps.

By the way, you are wrong about the RSS increasing from the deallocation.

  ...
  print "\nbefore deallocating the huge hash\n" . qx {ps -o rss $$};
  undef %x;
  print "\nafter deallocating the huge hash\n" . qx {ps -o rss $$};
  ...
[download]

Initial size:
  RSS
 1684

after allocating a huge hash:
  RSS
 9088

before deallocating the huge hash
  RSS
12532

after deallocating the huge hash
  RSS
12020

after the huge hash goes out of scope
  RSS
12020
[download]

The increase you saw was probably used to hold the results of keys %x.