in reply to PSQL and many queries

Do it the COBOL way:

  1. Sort the file in ascending order by name, using a disk-based sort. (Don't worry... the operation is uncommonly fast.)
  2. Query the database, ORDER BY name. If you like, go ahead and extract this into another flat-file.
  3. Now you are presented with two identically sorted data-streams. The operation that you are now looking to perform is called a merge.

It is possible to perform this operation (of course, without writing any new code to do so yourself...) by processing the two files sequentially. No searching is required. “10 million records?” No problem!

(If you ever watched old sci-fi movies and wondered “what all those spinning tapes were doing” ... now you know.)