|
Re: how to write < 10K files to folder
by kennethk (Abbot) on Feb 09, 2009 at 18:29 UTC
|
The best logic is to break the main folder into a series of subfolders based upon some logical criterion. It's usually done with some logical portion of the name. For example, if files have date-like names (2009-02-09.log), you could use the path 2009/02/2009-02-09.log. Without knowing more about your problem, I can't really say more.
Update: I was looking for Efficient processing of large directory, but apparently MidLifeXis's Super Search Foo is greater than mine. Definitely a good thread to read. | [reply] |
Re: how to write < 10K files to folder
by MidLifeXis (Monsignor) on Feb 09, 2009 at 18:49 UTC
|
When dealing with quantities of this amount, you probably want to know the behaviors of your file system.
To continue on kennethk's post, you should also make a decision based on your type of file system. 10K files can be too many files for a standard UFS file system to stand up to (because of needing to read more blocks for a directory than can be handled by a single read). One email processing system uses a directory size of 23 (IIRC) to increase its performance because of this. In the case of 1 million files, you would need to go 4 directories deep to handle 1,000,000 files in a small number of reads.
See Re: Efficient processing of large directory and Unix_File_System for more information.
| [reply] |
Re: how to write < 10K files to folder
by BrowserUk (Patriarch) on Feb 09, 2009 at 18:59 UTC
|
for /l %i in (1,1,20000) do @echo .>%i.txt
And it took about 6 seconds to create 20,000. Not that I'd advise actually doing that though.
Why do you need to create a million files? How big are they? What do their names look like?
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply]
[d/l] |
Re: how to write < 10K files to folder
by ikegami (Patriarch) on Feb 09, 2009 at 19:03 UTC
|
Just create each file in a different folder :)
Seriously, your question is not complete.
| [reply] |
Re: how to write < 10K files to folder
by brycen (Monk) on Feb 09, 2009 at 19:02 UTC
|
As others mentioned, break it logically (e.g. "a/aaron b/bryce").
Or, you could archive files before storage, with something like zip or tar. It all depends on the size of the files and how easy or quick access needs to be. | [reply] |
Re: how to write < 10K files to folder
by hbm (Hermit) on Feb 09, 2009 at 19:07 UTC
|
| [reply] |
|
|
| [reply] |
|
|
Kudos from kenneth,
for Super Search Foo;
I compliment your
Title::Parser too!
But "less than" points left,
and programmed sans heft:
A simple last is fine,
if nine-nine-nine-nine.
It's writing more than
that's made OP wan.
Millions is his plight;
Millions! To the right!
And folder misleads,
it's really folders --
"How to write million
files to subfolders"?
(sigh; ex pee depletes
as post gets older...)
| [reply] |
|
|
|
|
|
|