in reply to 15 billion row text file and row deletes - Best Practice?

No doubt you've probably already decided on a solution. In response to ideas that require a kill list loaded into memory (i.e. for example my grep suggestion): if one needs to remove 80GB of data due to the 300GB disk space constraint and there are 30 bytes per record and each serial number is 11 bytes. Then the amount of memory to load all those serial numbers into memory is: -

(80GB / 30B) * 11B = 29.3333333GB

(calculation courtesy of Google).

  • Comment on Re: 15 billion row text file and row deletes - Best Practice?