Re: Finding and sorting files in massive file directory

I have a directory with c. 2-3 million files (and growing) that I need to categorise and compress.

shell commands on the directory - find, ls etc - just hang!

Use wildcards in bash or glob in perl (ie: Fragment and move to several directories with mv a*.* /A-dir. Compress the files beginning for b with something like gzip b*.*)

Comment on Re: Finding and sorting files in massive file directory

Replies are listed 'Best First'.
Re^2: Finding and sorting files in massive file directory by CColin (Scribe) on Jan 20, 2013 at 21:30 UTC
Seem to recall when I tried something like this it failed due to too many arguments being passed to the command. ie. you can't run a million or so files through gzip command?	[reply]
Re^3: Finding and sorting files in massive file directory by frozenwithjoy (Priest) on Jan 21, 2013 at 01:04 UTC
If you put something like this in a while (there are files that start w/ `a`) loop, you should get around the issue of too many arguments. (Be sure to set `increment=1` before the loop). mkdir dir_a.$increment mv `ls a* \| head -n1000` dir_a.$increment/ let increment++ [download]	[reply] [d/l] [select]
Re^4: Finding and sorting files in massive file directory by choroba (Cardinal) on Jan 21, 2013 at 02:04 UTC
The problem is you will get "too many arguments" from `ls a*` already. لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ	[reply] [d/l]
Re^3: Finding and sorting files in massive file directory by MidLifeXis (Monsignor) on Jan 21, 2013 at 13:12 UTC
See the xargs command. It can help with breaking apart large command lines. `find . -type f -name b\* -depth -2 \| xargs $command_that_can_be_run_mu +ltiple_times` [download] Update: added example. --MidLifeXis	[reply] [d/l]