I think you don't want to load the data all at once into memory and then do the analysis. I think you should analyse the data one by one, create temporary results, and then work on that result. From my experience, this is a better way of working with large data set.