The OP is, in effect, making two allocations with a free between them. The first allocation is a very large AOH that causes Perl to allocate very near 2 GB. That allocation is then "freed" by undefing the array containing the hashes. Then a largish array is allocated containing integers. It is in the second phase that I infer things are going pear shaped for the OP, although I can't reproduce that behavior.

Perl doesn't generally free memory back to the system, but the space released following the first large allocation should be available for the second allocation however.