Re^2: Not sure what I am doing wrong

Thanks for the reply, I guess its not so much a problem as I think there maybe a better way to search through the logs for each keywork(or hoping there is). The problem is that each log file is about 10-18mb in size compressed so searching for each 100 keyword against every single line in the log file takes a really long time. Takes about 40-50 minutes to search each log file for the 100 keywords. Not sure if there is a way to search faster?

Comment on Re^2: Not sure what I am doing wrong

Replies are listed 'Best First'.
Re^3: Not sure what I am doing wrong by DStaal (Chaplain) on Nov 24, 2008 at 16:28 UTC
40-50 minutes per log file? Yeah, you should be able to speed that up... (Dependent on hardware of course, but my Mail::Log::Parse modules go through 8-9MB (after compression) logfiles in around a minute per (compressed) megabyte, and they do a lot more work than this.) The other comments on combining patterns could be useful, but the other thing to look at is what exactly your patterns are. Small differences in patterns can make some fairly major differences in speed. (Of course, if you are doing literals it's probably not that bad.)	[reply]

Replies are listed 'Best First'.

Re^3: Not sure what I am doing wrong
by DStaal (Chaplain) on Nov 24, 2008 at 16:28 UTC

40-50 minutes per log file? Yeah, you should be able to speed that up... (Dependent on hardware of course, but my Mail::Log::Parse modules go through 8-9MB (after compression) logfiles in around a minute per (compressed) megabyte, and they do a lot more work than this.)

The other comments on combining patterns could be useful, but the other thing to look at is what exactly your patterns are. Small differences in patterns can make some fairly major differences in speed. (Of course, if you are doing literals it's probably not that bad.)

[reply]