Re^2: 15 billion row text file and row deletes

Would it be faster to do relative seeks? I'm just thinking that toward the end of the file you're seeking across 300+GBs of data every time you delete a single line. would

seek $fhRead, $r2 - $w2, 1;
[download]

work?

Just a thought...

Edit: While it's obvious, i think it's also worth noting that if each record is the same size (which seems likely) then it can be done in one pass instead of two (or at least, one pass for each chunk of kill file, depending on kill file size).

Comment on Re^2: 15 billion row text file and row deletes - Best Practice? Download Code

Replies are listed 'Best First'.
Re^3: 15 billion row text file and row deletes - Best Practice? by BrowserUk (Patriarch) on Dec 01, 2006 at 23:20 UTC
Would it be faster to do relative seeks? It would, but I think almost all modern implementations of seek would automatically translate an absolute seek into a relative seek anyway. I don't know for sure, but I find it hard to believe that an absolute seek would ever cause the OS to 'go back to the beginning of the file'. In reality, seeking simply adjusts a number somwhere and it's not until the next read or write that it has any affect at all. The third parameter to seek is simply a throwback to when files stored on tape were commonplace. With regard to the possibility that the file has fixed length records, you're right it would be much easier and faster if that were the case. But from the (scant) information provided by the OP, it seems unlikely. Whilst he shows the serial number padded with leading zeros, he also shows a name field, which is empty in the given example, but names vary in length, so unless they are all empty, it probably indicates that these are variable length records. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal? "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]

Replies are listed 'Best First'.

Re^3: 15 billion row text file and row deletes - Best Practice?
by BrowserUk (Patriarch) on Dec 01, 2006 at 23:20 UTC

Would it be faster to do relative seeks?

It would, but I think almost all modern implementations of seek would automatically translate an absolute seek into a relative seek anyway. I don't know for sure, but I find it hard to believe that an absolute seek would ever cause the OS to 'go back to the beginning of the file'.

In reality, seeking simply adjusts a number somwhere and it's not until the next read or write that it has any affect at all.

The third parameter to seek is simply a throwback to when files stored on tape were commonplace.

With regard to the possibility that the file has fixed length records, you're right it would be much easier and faster if that were the case. But from the (scant) information provided by the OP, it seems unlikely. Whilst he shows the serial number padded with leading zeros, he also shows a name field, which is empty in the given example, but names vary in length, so unless they are all empty, it probably indicates that these are variable length records.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

[reply]