Is there any reason why you haven't moved the data into a (pick your flavour) database? Searching data in efficient ways is what SQL engines are really good at.