Quickly Find Files with Wildcard?

expresspotato has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Quickly Find Files with Wildcard? by Albannach (Monsignor) on Mar 23, 2009 at 01:14 UTC
I would try using substr instead of invoking a full regex since your match string is a constant. Secondly, I hope you drop out of your foreach loop once the file you want has been found (unless you expect more than one match that is), otherwise you will continue you expensive test for nothing. Hope this helps! -- I'd like to be able to assign to an luser	[reply]
Re: Quickly Find Files with Wildcard? by ysth (Canon) on Mar 23, 2009 at 01:01 UTC
How long is too long? How many files are there? You could try the locate command. If that's still to slow, you may need a different filesystem. (Or better yet, a different organization of your files.) -- Online Fortune Cookie Search Office Space merchandise	[reply]
Re^2: Quickly Find Files with Wildcard? by expresspotato (Beadle) on Mar 23, 2009 at 01:13 UTC
Thank you for the prompt reply. The query takes about 1-2 seconds per server. Considering this is running on 10 severs for example, without threads it could take up to 20 seconds for a full query. I do use threads and have been able to get it down to about 5-8 seconds, but again this is quite slow. Also locate looks very fast, however I'm unsure how to prevent it from recursing.	[reply]
Re^3: Quickly Find Files with Wildcard? by ysth (Canon) on Mar 23, 2009 at 01:22 UTC
Where does the extra time between 1-2 seconds per server vs. 5-8 seconds for a threaded lookup on all servers come from? Are you doing communication between servers, or are the filesystems from all the servers available locally? If the later, is nfs or whatever actually slowing things down compared to a solution using ssh? To "prevent" locate from recursing, filter the results looking for your specific directory. -- Online Fortune Cookie Search Office Space merchandise	[reply]
Re: Quickly Find Files with Wildcard? by linuxer (Curate) on Mar 23, 2009 at 01:21 UTC
Just some thoughts to this topic. Did no testing or benchmarking. So all of this could be nonsense ;o) How many entries are read? Maybe it is a speedup if you grep for plain files when reading from the directory (so you have less entries in `@a` which are checked afterwards). This maybe only makes sense, if there are much more directories than files. `sub find_hash_core { my $dir = './'; opendir my $dirh, $dir or die "$dir: $!\n"; # if you need the current workdir, see Cwd how to retrieve that and +restore it later chdir $dir; my @files = grep { -f $_ } readdir $dirh; for ( @files ) { if ( m/1234yourhashvalue/ ) { print "found hash"; }` [download] You could even check that hash value inside the grep: `my @files = grep { -f $_ && m/1234yourhashvalue/ } readdir $dirh;` [download] Did you try glob() instead of readir? Don't know if there is a big difference between those implementations... `chdir $dir; my @files = glob( ".1234yourhashvalue" );` [download] Did you try the linux ~~`file`~~ `find` command yet? `my @files = qx{ find $dir -maxdepth 1 -type f -name ".1234yourhashval +ue" };` [download] I think for a more detailed answer, please give more information... Otherwise it's up to you to find a solution... Update: fixed file/find typo	[reply] [d/l] [select]
Re^2: Quickly Find Files with Wildcard? by expresspotato (Beadle) on Mar 23, 2009 at 02:32 UTC
Thank you for all your suggestions. The speed of responses here is extraordinary! It seems accessing these remote file systems over SSHFS was the main bottle neck. Also exiting on the first find of the hash seems to have helped reduce query times. Now the requests are still made using threads, but to each server directly by calling a specially formatted webpage that will return if that server has the hash key in question. My hats to you Monks!	[reply]