in reply to Re^8: Working with large amount of data
in thread Working with large amount of data

I'm not completely sure about that. I had some bad experiences with Linux and disk drives a decade ago that have left me suspicious of how good the OS is at caching and reordering stuff. Things are certainly better now, but how much better I do not know.

Put it this way. If I was solving this problem on this hardware, I'd be sure to do some trial runs on smaller sets. And one thing I'd be testing is how many pieces to split a file into in one pass. Because it could matter.

  • Comment on Re^9: Working with large amount of data