in reply to Memory Question

As derby points out, you don't need both hashes in memory at once. Further than that, you could use a tied hash.

Another approach is to keep the files sorted and then use the comm command to find the differences. E.g.

comm -23 old new >oldonly comm -13 old new >newonly