use Digest::MD5 qw(md5_hex);
my $name = "123456.jpg";
my $path = md5_hex($name);
$path =~ s#^(.)(.)(.).*#$1/$2/$3/$name#;
[download]

We decided to use md5 over the other options. Some reasons included:
• md5 creates a balanced tree. With the numerical way, directories would vary from having 1 to 10,000 files in them, assuming 3 levels of directories in the form "1/2/3".
• IDs of less than 3 characters get treated the same way. (With the numerical version, files ended up in higher level directories, or got padded with zeros).
• It's still not very hard to find a directory "by hand" if a programmer needs to do that. Using md5 on the ID on the command line will quickly reproduce the result. This would work as well:

  find ./uploads_dir -name '1234.jpg'

  I also found out we need to plan for more like 1.5 million images.

  Mark Stosberg

One difference is that takes the prefix of a file name, allowing the suffix to be of arbitrary length:

123.jpg -> /1/12/123.jpg
114884.jpg -> /1/11/114884.jpg
[download]

The digits in the end of the number are more likely to be uniformly distributed:

123.jpg -> /3/23/123.jpg
114884.jpg -> /4/84/114884.jpg
[download]

Do you want a million files in a directory, and want to create them fast? No problem.

I've used ReiserFS in one situation in the past where I just copied a tree structure of about 70Gb onto a ReiserFS file system: in the end, it only used about 55 Gb. Never had any trouble at all. Highly recommended!

Liz

I agree. Reiser does an excellent job of handling the many-files-in-one-dir problem, avoiding the need for artificial hashing schemes.

Regarding reliability, I used reiser3 now exclusively on my 2.6.2 laptop and have done every murderous thing possible to it including hard poweroffs in busy-writes and numerous attempts to get software-suspend working (got it in the end but did many horrible things to my system in the process, reiser took it all without blinking).

I understand that there were problems in the past, and I would highly recommend that solid testing be done prior to deployment on a production platform, but I really do think the solution superior to the directory hashing workaround if the option is available.

Interesting to know. I've personally had bad experiences with ReiserFS, at least as provided by the Mandrake Linux 9.2 installer. On my own machine I got crippling file system errors after just a few days with it. I quit using it at that point. Another friend was using ReiserFS and found it hung at boot time after a few days, needing a manual fsck. I've had less problems with 'ext3'.

Mark Stosberg

There probably IS a better way. But is it worth it?

I'd suggest you set up a simple isolation layer, so you can do

$fh=image_open("123456.jpg");
[download]

If it's fast enough, then don't worry about it. If you have a performance issue, then worry about making it better.

Offhand, I suspect if you have a performance issue, it will come from the horrible impact this kind of structure will have on your disk caching, since accessing any given file is going to involve reading 4 directories then the file, and your images may not be accessed in any kind of cache-friendly order.

If it turns out the performance bites, you can change your open routine to use some other underlying structure without affecting the rest of the code...

Hi Mark.

Check out Re: Efficient processing of large directory, a previous node I wrote dealing with a very similar question. The solution is very implementation specific, and is based on what your filesystem's characteristics are.

--MidLifeXis

P.S. Is this an upgrade to your C::A image storage app?

Yes, it's likely this code will wind up in a version of CPAN:CGI::Uploader. The most current version has a function called 'build_loc' to handle the path name generation. So it may be possible to support more than one storage scheme easily, or at least sub-class the module and provide your own if you don't like mine.
Mark Stosberg

1/2/3/123.jpg
8/6/7/8675309.jpg
[download]

my $fullpath = $file; # Not just the number, but e.g. "7893.jpg"
$fullpath =~ s|(\w{1,3}).*|join('/',split(//,$1),$&)|e 
  or die "Bad filename $file";
[download]

On a small scale (1 initial letter), this is the way the terminfo database files are stored, and is also pretty much how Debian organizes their packages on their ftp sites.

But I'd personally go with the ReiserFS suggestion myself, assuming you have full control of the target box.

While we don't store as many images as you are talking about (usually around the 20K+ mark), we have 12TB of info that we deal with on a regular basis. Our scheme for keeping track of these images is splitting accross the different archives (RAID partitions) which are used as seperate directories, all linked to one central directory. From there we give each separate image a unique 10 character image code and use that as the name of the sub-directory. The image is then stored within that sub-directory. All of this is kept track of in an Oracle database.

In the *nix world it would look something like this:

/base_dir/images_archive1/<10_digit_unique_id>/image/<image_name>

Now this is a very simplified version of how our archive is set up, but it does work fairly well. I think that the major player here is the database keeping everything straight. The previous solutions using the 10K per archive sounds like a good cutoff point to me, also. I know using to many image sub-directories will probably cause you problems.

I know that this is not a Perl solution, but maybe it helps in some little way.