Re^2: 15 billion row text file and row deletes

Sorting 15 billion rows of text will be non trivial in terms of time and/or memory ;). I would drop the deleted SNs as you find them though.

--Brig

Comment on Re^2: 15 billion row text file and row deletes - Best Practice?

Replies are listed 'Best First'.
Re^3: 15 billion row text file and row deletes - Best Practice? by OfficeLinebacker (Chaplain) on Dec 03, 2006 at 17:03 UTC
Brig, You're right. gzipping would be faster than sorting, though, right? Worth it? The thought of using DBD::CSV also crossed my mind. That would be using a DB interface to a CSV file. Don't know if that meets the OP's requirement not to use a DB. Great topic. ++ _________________________________________________________________________________ I like computer programming because it's like Legos for the mind.	[reply]

Replies are listed 'Best First'.

Re^3: 15 billion row text file and row deletes - Best Practice?
by OfficeLinebacker (Chaplain) on Dec 03, 2006 at 17:03 UTC

You're right. gzipping would be faster than sorting, though, right? Worth it?

The thought of using DBD::CSV also crossed my mind. That would be using a DB interface to a CSV file. Don't know if that meets the OP's requirement not to use a DB.

Great topic. ++

_________________________________________________________________________________

I like computer programming because it's like Legos for the mind.

[reply]