Maybe ngram_counts is leaking memory.

As long as you store $ngram_href somewhere, I wouldn't expect it to get garbage collected.