in reply to Memory usage while tallying instances of lines in a .txt file

How many lines is the input file?

columns 2 and 3: are they always integers; do they have any well-defined min and max possible values; is their disturbution sparse (e.g. is it possible col 3 might have values 336158, and 336159, but then nothing till 491797, or is it likely that most of the gaps inbetween will appear at some point)?

Dave.

  • Comment on Re: Memory usage while tallying instances of lines in a .txt file

Replies are listed 'Best First'.
Re^2: Memory usage while tallying instances of lines in a .txt file
by TJCooper (Beadle) on Dec 05, 2016 at 17:23 UTC
    The input files can contain millions of lines - with around 1-1.5m unique entries that will be tallied up. The entries are indeed sparse.