Re^2: Design flat files database

1. I am trying to find the sweet spot between directory access time and the number of levels.
The folders hierarchy will be automatically created on the fly.
I mean if there is only 2374 , so the last user will have folder 2/3/7/4/2374
The same about 2 digits hierarchy, if there is more that million of id's another level will be added automatically.

At that point it's not for specific filesystem, but if the system will grow up I will move it to server with the suitable filesystem.

2.Ok , for now I'm using ext3 but I will move towards something better if needed.

3.Yes , I'm building an facebook/google+ like online community project.
I have already finished one project, it's working fast but when I benchmarked it there was some limitations on fequests per second but the CPU wasn't on 100% and still few gb of free memory...
I think it because of HD I/O limitations.Because of that i want to understand how to make it super fast.

q:Will replies have their own IDs within the 6-digit ID space?
a:yes , the replies will have an uniq id

q:Will replies only be displayed subservient to their parent? Or will they be viewable individually?
a:there will be post->replies structure.
Each post will have a file for it's replies
They won't be viewable individually,but there maybe will be an option to edit or delete the reply .So if the replies for specific post will be stored in the same file , it will need to be rewritten in a case some one want to edit only 1 reply.

q:Are replies to replies possible?
a:no

1-q: already answered .
2-q: I have a good experience with flat files databases, I started to read a book about mysql optimization and at some point I was upset .
It's not that simple as it seems to be. There is many tricks that need to know and that come with experience, to much options that I don't need.
I just afraid that in some point I will get stuck with it or the performance will be poor at some point.

Thanks for the informative post

Comment on Re^2: Design flat files database Download Code

Replies are listed 'Best First'.
Re^3: Design flat files database by BrowserUk (Patriarch) on Jul 15, 2011 at 15:55 UTC
replies will have an uniq id... Each post will have a file for it's replies. There is a conflict there. If replies have uniq IDs, then their bodies will exist in the hierarchy, so what will you store in the "file of replies"? My suggestion would be to store symlinks to the replies in the directory holding the body of the parent. That way, you do not have to traverse the hierarchy from the root to find them (as you would if you store a file containing the reply IDs). Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]
Re^4: Design flat files database by AlfaProject (Beadle) on Jul 15, 2011 at 21:15 UTC
i didn't really understand , what i mean is : user dir:`us/er/user2/` posts file:`us/er/user2/posts.txt` replies folder:`us/er/user2/replies/` replies for post 12`us/er/user2/replies/12.txt` (txt extention is only for example) inside a post file posts stored as : `id1\|time\|post_content\n` `id2\|time\|post_content\n` `id3\|time\|post_content\n` if i want to read the last few post's , i will use a read backwards module, or something similar. the replies files have the same structure as post file only with name of the user replied. `id1\|time\|user\|reply_content\n` `id2\|time\|user\|reply_content\n` `id3\|time\|user\|reply_content\n` I just wanted to know what is better , to store it like this or to make a file for each reply/post ?	[reply] [d/l] [select]