Re^2: Fastest way to recurse through VERY LARGE directory tree

Unless the OP needs the entire list of "tens of millions of files", I'd suggest not using File::Find::Rule, and I'd instead suggest an iterator or callback based routine (like File::Find). If you can process files one at a time, there's no need to build such a huge list.

Comment on Re^2: Fastest way to recurse through VERY LARGE directory tree

Replies are listed 'Best First'.
Re^3: Fastest way to recurse through VERY LARGE directory tree by eff_i_g (Curate) on Jan 21, 2011 at 17:14 UTC
runrig, I don't follow you. File::Find::Rule does not simply return every file (although it can). You can instruct it what to return based on type, size, name, and even make a determination via a custom sub. You can have the sub perform actions and ignore the larger return value, or iterate with the `start` and `match` methods.	[reply] [d/l] [select]
Re^4: Fastest way to recurse through VERY LARGE directory tree by runrig (Abbot) on Jan 21, 2011 at 17:24 UTC
Read the OP. I didn't say FFR has to return the entire list, the OP says the entire list needs to be processed. So, in this case, FFR would return the entire list of files.	[reply]
Re^5: Fastest way to recurse through VERY LARGE directory tree by eff_i_g (Curate) on Jan 21, 2011 at 17:38 UTC
Ah, gotcha. But you can work around building the entire list: `use warnings; use strict; use File::Find::Rule; File::Find::Rule->file->exec( sub { print $_[2], "\n"; return 0; } )->in('/path');` [download]	[reply] [d/l]