1. As tall_man said, this is a typical situation of merge sort. You can try the module given by tall_man.

  2. The other approach is to do the merge sort by yourself.
    1. First sort each file needs to be merged, sounds like you have program for this already.
    2. Have your own merge sort program do this (for example you are trying to merge n files),
      1. open all n files you want to merge
      2. open output file for merged result
      3. read one line from each of those n files
      4. compare those lines, and write out the smallest one
      5. if a line has been written out in step 4, then read next line from that file, if eof, close that file.
      6. goto 4, unless all file closed, i.e. finished (there is some subtle thing here, make sure that after all file closed, you still write out all the left over lines already read in)

      This algorithm only holds n lines at the most, memory usage is carefully designed.


  3. Another approach is to have a single script running as log server, and all your monitors just send logs to that server thru UDP, in this way the log msgs would be in order from the beginning.

    I used this approach in a real telecomminucation system, and yet speed is not a problem.

    But you have to spend some time on the log server, carefully lay out things like block writing etc.

In reply to Re: Log parsing by timestamp dilema by pg
in thread Log parsing by timestamp dilema by Limbic~Region

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.