My machine is not maxed out. It is a dual proc system, with its filesystem being a hardware raid 0, and plenty of RAM. Theoretically I could use a DB, but then that raises the scenario of, read the data, sort the data into relevant chunks, now do I A) send the data to a DB of some sort, to later reopen, reread into memory what ever way, and then duly process it, or B) simply process it now. Also if I were to segregate the computation of the data across hosts, I'm still dealing with finding a sane way of splitting the data, sending it out to some other host, polling to see when they are done, or waiting for them to finish processing, and then pulling all the data back together and correlating it. All of which adds to the run time.

I have also realized that not all the events for a msgid will occur within a given logfile, due to rotation, but im not even gonna go into maintaining state across files, for a very small percentage of the actual data. As you can see in the code I'v been allowed to kinda punt in regards to absolute accuracy, even though it may drive my inner anal-retentive geek crazy, such is life

I guess I should step back and say, I'm not attacking your points, so much as stating I've already considered it and didn't think the benefits out weighed the returns in terms of code logic and run time. No offense intended in any way shape or form. I should also add that the system doing the processing itself is not one of the MTAs. It is a completly seperate host, which is doing just about absolutely nothing aside from sshd. Its also a FreeBSD box if it makes any difference

/* And the Creator, against his better judgement, wrote man.c */

In reply to Re: Re: The need for speed by l2kashe
in thread The need for speed by l2kashe

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.