in reply to Re^3: Design flat files database
in thread Design flat files database
Well, my own personal experience has led me to develop this one rune: “Do not, if possible, store BLOBs in an SQL database of any kind.” I know that this is an oversimplification; that there must be SQL databases that do this really well and that don’t become an insufferable management PITA. (As the Moody Blues song says, "I know they’re out there some-where ... some-where ... sommmme-where ...”) Whereas, I also know of another commonly-used type of database that is ideally suited for storing tens of millions of variable length records: it’s called “a file system,” and the records are called files. It’s not a good index, but it’s a great store.
Knowing, as I do, that I am about to create perhaps millions of very similarly named files, and also knowing that this “concentration of weight in a small area” might create a PITA of a different sort for the filesystem or for me or both, I would create an additional tree-taxonomy in the form of a directory structure of my own choosing. But, frankly, I would not spend too much time searching for that “sweet spot.” I would hazard that now I am starting to try to meddle in the filesystem’s proper business, probably with no practical return on the time-and-effort investment. Instead, I would simply arrange things so that I can change things later, if I need to.
A more interesting concern might be how to perhaps distribute the data among multiple physical volumes, by assigning different high-level directories to different physical volume-groups. (Assuming of course that I am not paying mega-bucks a month for a system like Oracle.) This taxonomy would let me do that.