I think these folks are on the right track. The hash is only built for the line being analyzed. Evidently the number of items on a line is small compared with the 3 million line limit.

As one hint for HUGE hashes (this situation doesn't qualify from what I understand), but the default hash starts with 8 key "slots". As the hash gets larger, 8,16,32,64,128,etc, all hash keys in the hash have to be re-calculated. If you start getting into hash sizes of like 100,000 keys, this doubling process can cost. It is possible to start a hash with a larger number of buckets than the default of 8, by assigning a scalar value to keys, like keys %hash=2**16 or whatever. If the hash is less than 1024 total key space, this usually doesn't make much difference. Good values for number of available keys is 1/2 expected total number of items in hash (a place to experiment from).