Multiple regex against files with speed

crabbdean has asked for the wisdom of the Perl Monks concerning the following question:

Hi Fellow Monks and Nuns,

I'm rewriting a perl module to do file listing because quite frankly I don't like File::Find. I've got it listing beautifully, its fast and in OO. I want to now add features. One is to be able to specify types of files or directories to match against.

In essence you have to be able to specify many types of files (eg. ~*.doc, ~*.tmp etc). It then has to check each file against ALL these types. Simply I want to know, what is the fastest way of doing this?

So far I'm considered:
1. Storing the regex string into the values of hashes and then doing a regex against each of the search hash values.

$found = 0;
$hash{1} = '*.doc';
$hash{2} = '*.xls';
$hash{3} = '*.ppt';
map {
     $found = 1 if $file =~ /$_/;
} values(%hash);
[download]

2. OR stringing all the searchstrings together into one big regex

$found = 0;
$hash{1} = '*.doc';
$hash{2} = '*.xls';
$hash{3} = '*.ppt';
for (values(%hash)) {
     $string .= $_ . "|";
}
$found = 1 if $file =~ /$string/;
[download]

Any other ideas? Speed is the key here. Thanks.

Oh, and if you have any other requests for what you'd like in this module or frequent types of things you do against files in your scripts, I'll see what I can do to add them in.

Dean
The Funkster of Mirth
Programming these days takes more than a lone avenger with a compiler. - sam
RFC1149: A Standard for the Transmission of IP Datagrams on Avian Carriers

Comment on Multiple regex against files with speed Select or Download Code

Replies are listed 'Best First'.
Re: Multiple regex against files with speed by jeffa (Bishop) on Apr 28, 2004 at 13:11 UTC
How about not reinventing a perfectly round wheel and adding File::Basename to your problem? Or even better --- have you seen File::Find::Rule yet? `my @files = File::Find::Rule ->file() ->name( '.doc', '.xls', '*.ppt' ) ->in( @DIRS ) ;` [download] UPDATE: crabbdean just pointed out File::Find memory leak to me. jeffa L-LL-L--L-LL-L--L-LL-L-- -R--R-RR-R--R-RR-R--R-RR B--B--B--B--B--B--B--B-- H---H---H---H---H---H--- (the triplet paradiddle with high-hat)	[reply] [d/l]
Re: Re: Multiple regex against files with speed by crabbdean (Pilgrim) on Apr 28, 2004 at 13:35 UTC
The module I've build already breaks the full file name into 4 pieces - a relative path, base directory, fullfile name and filename. So I've got that bit nicely. The problem with the "File::Find::Rule" is that it utilises the "File::Find" which I'm trying to avoid. Unfortunately considering "File::Find" is a core Perl module it means re-inventing the wheel. Ho hum! Although I do like the interface used on some of the "File::Find::Rule" methods. I have a read through his code. Dean The Funkster of Mirth Programming these days takes more than a lone avenger with a compiler. - sam RFC1149: A Standard for the Transmission of IP Datagrams on Avian Carriers	[reply]
Re: Multiple regex against files with speed by eserte (Deacon) on Apr 28, 2004 at 14:15 UTC
If you're for speed, then take a look at Regexp::Optimizer. This will append regexes into one regexp and additionly will do trie optimization. Assuming you really mean regexps, your example code uses file globs instead.	[reply]
Re: Multiple regex against files with speed by pelagic (Priest) on Apr 28, 2004 at 14:13 UTC
Benchmark your solutions. There are plenty of Benchmark exaples around here. pelagic	[reply]
Re: Multiple regex against files with speed by Roy Johnson (Monsignor) on Apr 28, 2004 at 14:34 UTC
I'm curious about why you're using a hash as an array instead of using an array. Note that you'll need to convert your glob patterns to actual regexes if you want to do regex matching. Stringing the regexes together is more naturally done with join than with a foreach loop and concatenation. The PerlMonk `tr///` Advocate	[reply]
Re: Multiple regex against files with speed by Fletch (Bishop) on Apr 28, 2004 at 13:19 UTC
`perldoc -f grep`, and read about `qr//` in `perldoc perlop`.	[reply] [d/l] [select]