To further the above: Think database. The work process you want is 'Read/compute/store' then 'query/display'. (This assumes you don't want exactly the same statistics every day, in which case just generate a static web page with that when the log files roll over.)

Databases work with massive amounts of data quite well: I deal with multiple logs that are about 1GB compressed every day, and once they are parsed and in the database, data return is nearly instant.

And, luckily enough, Perl has good database support for just about any database you'd ever want to use (and a few you wouldn't).

In reply to Re^2: Dealing with huge log files 100 MB to 1 GB by DStaal
in thread Dealing with huge log files 100 MB to 1 GB by gchitte

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.