Re^7: Working with large amount of data

Whenever you're considering a design remember the following. Seeks to disk cost 0.01 seconds each, but it is reasonable to stream data back and forth at 50 MB/s. Therefore it is worth doing a lot of extra work to be able to stream data rather than seeking to disk.

If you've done things right, for large data sets your time is entirely dominated by the time to stream through data. So 1 file vs 100 files is irrelevant. But splitting directly into 100 files may be a horrible idea for the simple reason that disk drives are typically able to stream data at high rates to a fixed number of locations. Like 4 or 16. So you'd probably want to split the data in multiple passes if you went with this design. (That is not to say that this is the right design. Personally I head in the merge sort direction rather than using hashing.)

As for going to a database, my experience is that when your data sets are near the capacity of the machine, databases often will run into resource constraints and not figure a way out. It isn't that the query runs painfully slowly, it is that it grinds away for several hours then the query crashes. That is one of the prime reasons that I have needed to do end runs around the database when working with large data sets.

Comment on Re^7: Working with large amount of data

Replies are listed 'Best First'.
Re^8: Working with large amount of data by salva (Canon) on Sep 22, 2009 at 07:53 UTC
But splitting directly into 100 files may be a horrible idea for the simple reason that disk drives are typically able to stream data at high rates to a fixed number of locations. Like 4 or 16 Are you completely sure about that? AFAIK the OS and the file system layer should mitigate any hardware limitation like that. Writes are cached and reordered before sending them to the hardware, so it shouldn't be any difference between writing to a file or to a thousand... Well, unless you have your file system configured to commit everything immediately, but this is not common because of the huge performance penalty it imposes!	[reply]
Re^9: Working with large amount of data by tilly (Archbishop) on Sep 22, 2009 at 15:13 UTC
I'm not completely sure about that. I had some bad experiences with Linux and disk drives a decade ago that have left me suspicious of how good the OS is at caching and reordering stuff. Things are certainly better now, but how much better I do not know. Put it this way. If I was solving this problem on this hardware, I'd be sure to do some trial runs on smaller sets. And one thing I'd be testing is how many pieces to split a file into in one pass. Because it could matter.	[reply]