I would strongly advocate a database.

Typically, you (or management) initially think this is a small one-time thing - all you need are a few summary numbers. What usually happens is that after you see a few nice, somewhat meaningful numbers, you realize what you are NOT seeing, and need to generate more stats. Then meta stats, then trends. Alarms/threasholds. And it goes on ...

WIth a database, you can store periodic info about an IP address - how frequently has this been in the top ten .. is his traffic rate increasing over time .. Which IP's have stopped being used. which subnet generates the most bad traffic etc...

The other advantage is that your data collection and analysis become separate processes. Now you can get close to real-time data analysis. Web interfaces become possible. It's all good (Except the work required to code all this, but even that can be enjoyable).

    ...each is assigned his own private delusion but he cannot see the baggage on his own back.

In reply to Re: Creative sorting and totalling of large flatfiles (aka pivot tables) by NetWallah
in thread Creative sorting and totalling of large flatfiles (aka pivot tables) by dbg

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.