To be absolutely sure your memory gets released back when you want, fork the code off. See Re: undefining hashes to free memory for more info on non-forked problems with memory.
Otherwise, the Perl interpreter will greedily hang on to the memory, figuring if you used it once, you probably will want to use it again.