comment on

i'm writing a log parser, and am trying to optimize the way i match valid entries in the logfile. its a logfile for a multiplayer fps (F.E.A.R.), and its not as simple as most logs ive worked with, because the data i want (info for a given player after each round) spans multiple lines. this is a sample log for a single game:

[Thu Sep 21 17:48:38 2006] 
[Thu Sep 21 17:48:38 2006] ------------------------------------------
[Thu Sep 21 17:48:38 2006] Server started.
[Thu Sep 21 18:37:22 2006] Client connected: Alpha
[Thu Sep 21 18:37:31 2006] Client connected: Bravo
[Thu Sep 21 18:38:20 2006] Client connected: Charlie
[Thu Sep 21 18:39:18 2006] Client connected: Delta
[Thu Sep 21 18:53:36 2006] 
[Thu Sep 21 18:53:36 2006] *** Results for Map: Worlds\ReleaseMultipla
+yer\Bypass
[Thu Sep 21 18:53:36 2006] 
[Thu Sep 21 18:53:36 2006] Team: Team 1
[Thu Sep 21 18:53:36 2006] Score: 93
[Thu Sep 21 18:53:36 2006] 
[Thu Sep 21 18:53:36 2006] Player: Alpha (uid: ad7023b7f46271acd31e1bd
+287613b6d)
[Thu Sep 21 18:53:36 2006] Score: 55
[Thu Sep 21 18:53:36 2006] Kills: 14
[Thu Sep 21 18:53:36 2006] Deaths: 15
[Thu Sep 21 18:53:36 2006] Team Kills: 0
[Thu Sep 21 18:53:36 2006] Suicides: 0
[Thu Sep 21 18:53:36 2006] Objective: 0
[Thu Sep 21 18:53:36 2006] 
[Thu Sep 21 18:53:36 2006] Player: Bravo (uid: 5fdcc95043dc4dac9d7b4af
+b8469eb4f)
[Thu Sep 21 18:53:36 2006] Score: 38
[Thu Sep 21 18:53:36 2006] Kills: 11
[Thu Sep 21 18:53:36 2006] Deaths: 17
[Thu Sep 21 18:53:36 2006] Team Kills: 0
[Thu Sep 21 18:53:36 2006] Suicides: 0
[Thu Sep 21 18:53:36 2006] Objective: 0
[Thu Sep 21 18:53:36 2006] 
[Thu Sep 21 18:53:36 2006] Team: Team 2
[Thu Sep 21 18:53:36 2006] Score: 135
[Thu Sep 21 18:53:36 2006] 
[Thu Sep 21 18:53:36 2006] Player: Charlie (uid: e94839cae76debf1418ab
+9dfaa4c01e8)
[Thu Sep 21 18:53:36 2006] Score: 61
[Thu Sep 21 18:53:36 2006] Kills: 15
[Thu Sep 21 18:53:36 2006] Deaths: 14
[Thu Sep 21 18:53:36 2006] Team Kills: 0
[Thu Sep 21 18:53:36 2006] Suicides: 0
[Thu Sep 21 18:53:36 2006] Objective: 0
[Thu Sep 21 18:53:36 2006] 
[Thu Sep 21 18:53:36 2006] Player: Delta (uid: b2ea959c1b3fa5c35ef6a6e
+576cdf2af)
[Thu Sep 21 18:53:36 2006] Score: 46
[Thu Sep 21 18:53:36 2006] Kills: 10
[Thu Sep 21 18:53:36 2006] Deaths: 4
[Thu Sep 21 18:53:36 2006] Team Kills: 0
[Thu Sep 21 18:53:36 2006] Suicides: 0
[Thu Sep 21 18:53:36 2006] Objective: 0
[Thu Sep 21 18:53:36 2006] 
[Thu Sep 21 18:53:36 2006] 
[Thu Sep 21 18:57:47 2006] Client disconnected: Delta
[Thu Sep 21 18:58:01 2006] Client disconnected: Alpha
[Thu Sep 21 18:58:17 2006] Client disconnected: Bravo
[Thu Sep 21 18:59:03 2006] Client disconnected: Charlie
[download]

big and unwieldy, i know, and this is just one round. i have a real logfile that spans about a month, its huuuuuge. now, i have a working parser for it, using a search & replace regex that 'nibbles' away at the log:

note: since the server may be windows or linux, i'm explicitly matching both LF and CRLF newlines, hence the \x0D?\x0A at the end of the line


# loop through, 'nibbling' at logfile with every
# successful match & replacement
while(
  $log =~ s{
  \[ ([^\]]+?) \] \s+ Player\: \s+ ([^\x20]+?) \s+ \( uid: \s+ (\w+) \
+)  \x0D?\x0A
  \[ [^\]]+? \] \s+ Score: \s+ (\d+)  \x0D?\x0A
  \[ [^\]]+? \] \s+ Kills: \s+ (\d+)  \x0D?\x0A
  \[ [^\]]+? \] \s+ Deaths: \s+ (\d+)  \x0D?\x0A
  \[ [^\]]+? \] \s+ Team \s+ Kills: \s+ (\d+)  \x0D?\x0A
  \[ [^\]]+? \] \s+ Suicides: \s+ (\d+)  \x0D?\x0A
  \[ [^\]]+? \] \s+ Objective: \s+ (\d+)  \x0D?\x0A
  }{}ixms
) {

  # grab values from regex backreferences
  $timestamp = $1;
  $player    = $2;
  $uid       = $3;
  $score     = $4;
  $kills     = $5;
  $deaths    = $6;
  $teamkills = $7;
  $suicides  = $8;
  $objective = $9;
  
  # ...process log entry...
}
[download]

this works, its just slow and rather inefficient, not to mention it will likely be a beast to maintain.

i was trying to change to using a regular match with a /gc flag because it would be much more efficient, and it would also allow me to bypass the backreferences altogether:


# loop through, grabbing values for every
# successful match with the /gc flag
while(
  ($timestamp,$player,$uid,$score,$kills,$deaths,$teamkills,$suicides,
+$objective) = 
  $log =~ m{ \G .+
  \[ ([^\]]+?) \] \s+ Player\: \s+ ([^\x20]+?) \s+ \( uid: \s+ (\w+) \
+)  \x0D?\x0A
  \[ [^\]]+? \] \s+ Score: \s+ (\d+)  \x0D?\x0A
  \[ [^\]]+? \] \s+ Kills: \s+ (\d+)  \x0D?\x0A
  \[ [^\]]+? \] \s+ Deaths: \s+ (\d+)  \x0D?\x0A
  \[ [^\]]+? \] \s+ Team \s+ Kills: \s+ (\d+)  \x0D?\x0A
  \[ [^\]]+? \] \s+ Suicides: \s+ (\d+)  \x0D?\x0A
  \[ [^\]]+? \] \s+ Objective: \s+ (\d+)  \x0D?\x0A
  }ixmsgc
) {
  
  # ...process log entry...
}
[download]

per my understanding, it should match \G as the beginning of the string on the first time through, then match it as the end of the previous match from then on...however, it only ever matches one time.

am i missing something blatantly obvious? or is there even a cleaner approach that i havent thought of?

__________
Build a man a fire, and he'll be warm for a day. Set a man on fire, and he'll be warm for the rest of his life.
- Terry Pratchett

In reply to regex issues with /gc in log analysis... by EvanK

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		`&`
	<		`<`
	>		`>`
	[		`[`
	]		`]`

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.