Have you read the FAQ entry on this? Memory used and free-d by Perl is not released to the O/S (except in MacOS, if I recall correctly), so the apparent memory usage will not go down, but that memory is still available for Perl to re-use for other items.
-- Randal L. Schwartz, Perl hacker