in reply to (OT) should i limit number of files in a directory

If you need to do this in the filesystem, which merlyn makes good points against, then you definitely don't want to steal characters from the filenames as RMGir says.

You probably also want to use "/op/opus/opusco/opuscows" rather than just "/op/us/co/opuscows" too, for the same reasons. It's possible to rebuild the whole directory system based on just the file names as long as the file names are intact, but the directory names being salvageable as well will help when something goes wrong.

Is there any data in these files? Are these hash-named files the files you're checking for existence, or are you using them to track the existence of other files? 16**32 is much larger than 3 million, so I'm guessing these are hashes of other files.

Is this a tracking system to see if files have been inserted into a document management system? If so, you'll have issues if the documents are editable because the MD5 sum will change. You'd have to delete the hash for the old version before the edit starts and recreate it after the edit every time. It might be easier to store both the document and the hash for it in a database if you're doing something like that.

  • Comment on Re: (OT) should i limit number of files in a directory

Replies are listed 'Best First'.
Re^2: (OT) should i limit number of files in a directory
by leocharre (Priest) on Sep 11, 2008 at 17:50 UTC

    There is data in these files, yes. Some have a little bit (1k) some a lot (up to maybe 400k.. not much more).

    I am not using them to keep track of existance of other files.

    The data itself is of interest, in regards to the hash/ the digest /the filename of this metadata (with no structure).

    "Is this a tracking sys..." Yes and no. I am *expecting* for the document names (locations), hosts, and data to change. If that happens, then that document is no longer the same document- it's irrelevant. ( I know.. that's a very concise summary of what's up- That discussion is a very large and involved one. )