For some log files the time stamps aren't exactly in order, and this process might lose a few records.

You should be able to find out of your log files suffer from this problem in a unix shell with a combination of 'cut' to extract the date, piped to 'uniq', and see if each date only shows up once.

The issue is this -- there are a few webservers that will record the time that the request was sent, but they write to the log until after it's been sent. If you have long-running CGIs and small static documents being served from the same server, and the server is very busy, you can end up with records from one day being written out before ones from the previous day.

...

In general, though, this is a great suggestion, and even if you do suffer from this issue, you'll likely only lose minimal records.

In reply to Re^2: How to quickly parse a huge web log file? by jhourcle
in thread How to quickly parse a huge web log file? by dbmathis

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.