"I didn't see any mentions of files so I wasn't going to jump to the conclusion that "huge dataset" means some single huge file or even any files at all."

Sorry, the data is in a single tab-delimited text file. I'm doing something similar to a correlation coefficient on terms assigned to genes in a large spread to generate similarity. For this, I'm using a kappa statistic, which involves pairwise comparisons between all genes and all annotations, it is a very large matrix.

Thanks for the suggestions though. I'll try implementing them and get back to the thread