At a conference, I saw Tim Bray (father of XML) talk about managing large data sets, and he urged people to not overlook how darn efficient the filesystem, including the directory lookup code, has gotten in modern open source "unix" releases. Don't add a database or a substructure until you've proven its need through benchmarking.

-- Randal L. Schwartz, Perl hacker
Be sure to read my standard disclaimer if this is a reply.