Re: Need directory scheme to store 400,000 images

If you want to do it really simple, use a ReiserFS partition. From the front page blurb:

Do you want a million files in a directory, and want to create them fast? No problem.

I've used ReiserFS in one situation in the past where I just copied a tree structure of about 70Gb onto a ReiserFS file system: in the end, it only used about 55 Gb. Never had any trouble at all. Highly recommended!

Liz

Comment on Re: Need directory scheme to store 400,000 images

Replies are listed 'Best First'.
Re: Re: Need directory scheme to store 400,000 images by PhiRatE (Monk) on Apr 13, 2004 at 01:34 UTC
I agree. Reiser does an excellent job of handling the many-files-in-one-dir problem, avoiding the need for artificial hashing schemes. Regarding reliability, I used reiser3 now exclusively on my 2.6.2 laptop and have done every murderous thing possible to it including hard poweroffs in busy-writes and numerous attempts to get software-suspend working (got it in the end but did many horrible things to my system in the process, reiser took it all without blinking). I understand that there were problems in the past, and I would highly recommend that solid testing be done prior to deployment on a production platform, but I really do think the solution superior to the directory hashing workaround if the option is available.	[reply]
Re: Re: Need directory scheme to store 400,000 images by markjugg (Curate) on Apr 12, 2004 at 19:45 UTC
Interesting to know. I've personally had bad experiences with ReiserFS, at least as provided by the Mandrake Linux 9.2 installer. On my own machine I got crippling file system errors after just a few days with it. I quit using it at that point. Another friend was using ReiserFS and found it hung at boot time after a few days, needing a manual fsck. I've had less problems with 'ext3'. Mark Stosberg	[reply]


Welcome to the Monastery
	PerlMonks