Re: Efficient Way to Parse a Large Log File with a Large Regex

You can give this a try and see just how slow it is.

@list = map({ quotemeta "129.$_.125.123" } (0..255));

$regex_text = join('|', @list);
$re = qr[($regex_text)];
print $re, "\n";

while (<>) {
    if ($_ =~ $re) {
        print "$1\n";
    }
}
[download]

Update: This seems to be faster than hash method.

-- gam3
A picture is worth a thousand words, but takes 200K.

Comment on Re: Efficient Way to Parse a Large Log File with a Large Regex Download Code

Replies are listed 'Best First'.
Re^2: Efficient Way to Parse a Large Log File with a Large Regex by Dru (Hermit) on Apr 12, 2005 at 19:30 UTC
Thanks. This is what I came up with based on your code: `use warnings; use strict; my @ips = qw/192.168.2.1 ..../; @ips = map { quotemeta } @ips; my $regex = join('\|', @ips); my $re = qr[$regex]; while (<>) { print if /$re/; }` [download] I'm then calling it liks so: `tail -f fw.log \| /usr/local/scripts/parseips` [download] It's taking up quite a bit of resources, but not bringing the server to it's knees. Thank you for the other suggestions also. I'm going to come up with a more perm. solution based on one of these that does not require me to stare at a terminal.	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^2: Efficient Way to Parse a Large Log File with a Large Regex
by Dru (Hermit) on Apr 12, 2005 at 19:30 UTC

use warnings;
use strict;

my @ips = qw/192.168.2.1 ..../;

@ips = map { quotemeta } @ips;
 
my $regex = join('|', @ips);
my $re = qr[$regex];

while (<>) {
    print if /$re/;
}
[download]

tail -f fw.log | /usr/local/scripts/parseips
[download]

[reply]
[d/l]
[select]