Blink.. blink.. Did you say, “‘loads’ 1.5 GB of data?‘ For good-heaven's sake... into where?!?!?!

When you're dealing with a volume of data like this, absolutely the worst approach in the known Universe is to “load it into ‘memory’ and then access it randomly.” As a matter of fact, your very first mistake is to even attempt to access such a volume of data “randomly” ... at all!

First of all, in any modern computer system, “memory” is virtual. In other words, memory is a cached disk file, and any attempt to consider it in anything other than those terms is preordained for failure.

Come back with me to the days of yesteryear ... to the earliest days of computers, in fact to the days before digital computers. Let me re-introduce you to a world where searching was not necessary, albeit for the very-obvious reason that it could not be done.

Take the Unix sort command and try to sort that 1.5 gigabyte file. It'll take a few minutes but you just might be quite surprised at how little time it actually takes. (Study the options of sort just a little more and you might slash that time in half.)

Once a file is known to be sorted, it does not have to be searched. At least, not for the sort-key. And if you needed to have different versions of that file sorted by different keys, well, that would probably take a lot less time than the approach you're banging your head against right now!

When you know that a file is sorted, then you always know two very-important things:

• All occurrences of any key-value occur together.
• Therefore, when the last occurrence of key-value “X” has been seen, there are no more.
• If any “gaps” occur, it can be conclusively stated that there do not exist any records which can fill that gap... anywhere. You know this, without searching.

Forget all of “this threading nonsense.” I'm very sorry to say, re-think your entire approach! (Believe me, you'll be very glad that you did.) Think about it:   what possible good will it do to subdivide the CPU's time in new-and-creative ways, when it can be quite-conclusively shown that it is only the speed of the I/O subsystem that has any conceivable relevance upon the solution-speed of this problem?

I'm serious:   come to a full stop and read my posting over and over again about ten times. Then take the rest of the day off, come back Monday and read it another ten times. Wait patiently until you say... “ahhhh-h-h-h-h...” I mean absolutely no offense when I say... bring your sunglasses. When that proverbial “little light” comes on, it is quite bright.