Refactor part that eats memory
my %hash = get_hash();,
maybe with MLDBM (or DB_File), or DBD::SQLite... maybe just flatten so its simple => "hash \tof\tvalues"
DBM::Deep is a better choice than MLDBM these days, as access to arbitrarily deeply buried data is far more transparent. To immediately address the OP's concerns, it is far more memory efficient than MLDBM too!