Re: Multiple patterns match in a big file and track the counts of each pattern matched

I need to match the patterns in the 1GB+ log file ... but it takes long, close to 2 mins, for 3 patterns

So, profile it. How much of that 2 mins is just simply reading the massive logfile before you do anything with it? Only once you have determined that the pattern match is the slow part should you consider taking further action regarding it.

If the pattern match does turn out to be a big contributor to the runtime, use index instead since you are actually matching substrings and not regexen.

Comment on Re: Multiple patterns match in a big file and track the counts of each pattern matched

Replies are listed 'Best First'.
Re^2: Multiple patterns match in a big file and track the counts of each pattern matched by ansh007 (Novice) on Nov 28, 2017 at 11:35 UTC
It takes 10 to 11secs to parse the whole file. That's how I knew matching takes time. If I give only one pattern, it takes 1min+ and slowly it increases. I need to make sure, it doesn't take much memory or CPU and it must run within 30-35 secs.	[reply]