I have a system that checks if a file exists, if not, the file is created.
Easy enough.
Now, the file count will be at least 100,000 and potentially 3 million in 12 months. every *filename* is a md5 hex sum digest thus, it is 32 chars long and each is 16 possible chars.

Space is not an issue here. These are small text files. I'm on GNU/Linux using ext3 partitions.

I'm considering if I should have a hack to keep the directory file counts to a minumum.
For example, pause does this with http://backpan.perl.org/authors/id/L/LE/LEOCHARRE/, notice L/LE/LEOCHARRE (yes, that is a directory not a file, let's be flexible here).

so if the file in question (that i will read or create) is named 'opuscows', it will really reside in either o/op/opuscows ( or more interestingly.. op/us/cows , then the first dir would have 256 entries, and every level would also have another 256 entries (16*2 for xx/) )

This would help keep my directory entries lower than say, 3 million.

This hack will slow down looking and writing, by a little bit.

But maybe this is not needed. I will not be searching for files, or doing a dir listing operation. The file is there or not.

Is there a limit to how many files I should have in a such a directory? I read that "There is a limit of 31998 sub-directories per one directory,..." - but this does not make mention of files in general.

Please excuse my broken up discussion.

update

After discussion in this thread, I am using mysql to serve the data instead of using a regular filesystem.

I had some text entries that were larger than 1M, this caused a problem for me at first. The default maximum packet for mysql is 1meg. You must change the max_packet_size in your mysql config file. Likely in /etc/my.cnf, you would add a 'max_packet_size=5M' line (for example) and restart the server.

In reply to (OT) should i limit number of files in a directory by leocharre

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.