Not to ruin all your fun, but have you considered just writing out the dups and piping your output through 'sort -u'? The 'sort' implementation on my system is very memory efficient and reasonably fast.

-sam