Re: Is there a way to set max results on File::Find::Rule?

A user interface to File::Find::Anything and a million files to search sounds like a bad combination. Have you considered the efficiency of only running File::Find periodically, dumping the results into a database of some description, and then running the search on that DB rather than on the filesystem directly. Even a flat file database would be vastly quicker than recursing the directory tree on a regular basis.

Not suprisingly this is not a unique problem. In *nix land there is a program called (s)locate(1) which does exactly this task (recurse the tree, dump to DB, search the DB). File::Locate is a perl interface to it.

Cheers

Comment on Re: Is there a way to set max results on File::Find::Rule?

Replies are listed 'Best First'.
File::Locate and File::Locate::Harder (was Re^2: Is there a way to set max results on File::Find::Rule?) by doom (Deacon) on Dec 21, 2007 at 04:54 UTC
In general, I hestiate to recommend a module that I wrote -- and I'm afraid I've been going through an "over-engineering" phase of late -- but if you're interested in File::Locate you might want to at least take a look at the documentation for File::Locate::Harder. In particular, if you look at the discussion of internals all the way down in the major section titled "NOTES", you'll see a summary of my discoveries in the world of "locate" variants. Note that if you want to use File::Locate you need to know where your system's "locate" database is stashed, and that varies from system to system and isn't always easily discoverable (it appears to be common for "locate"/"slocate" and "updatedb" to rely on compiled-in defaults that aren't documented in the man pages).	[reply]
Re^2: Is there a way to set max results on File::Find::Rule? by leocharre (Priest) on Jan 02, 2008 at 14:36 UTC
Indeed, using *x slocate or indexing to a database is a good way to go. This is useful for a semi static or static set of data. Unfortunately, the files I am searching are a set of network mounts- multiple users, rapid changes. Maintaining a database of so many files would .. . require a machine solely devoted to reloading an slocate database- perpetually. This would simply add to the complexity of the network- it may not worth the negatives right now. At some point, the cpu expense of using real time disk 'find' methods will call for a separate machine to do as you mention. Very good comment, tachyon-II	[reply]