As someone who's recently had to solve this (well, slightly different: I didn't have to do real-time) problem at work: Reading over and over is going to be slow. Probably slower than what you want to deal with.

To actually get decent speed I finally moved the assembly of transactions into the database: I insert each line as I read it, letting the database record fill in as it gets information. (Note that you need some way to know which record each line is going to be in, and identify it uniquely over the course of at least a log file. Depending on the format of the logs, it may be worth keeping track of just the 'active' transactions.)

Doing this allowed me to process 1-1.5GB of a data a day in a couple of hours. (Despite having to use a temp database, since the DBD install for the main database is borked beyond repair.)

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.