You can certainly get more help in terms of algorithms and other ideas if yor post your problem with more details here. I think better algorithm is way to go. In the case like yours, for a momemnt forget the memory required, you would have to process that vast number of data with some further algorithm as well. Using the techniques like 'Genetic Algorithm' may also help you.

artist