Storing all the lines in all the logfiles seems the wrong way to go here. Both of your objectives seem more parse-as-you-go.

For printing the top x lines, simply read the top x lines from each file, then sort them and print the top x of the sort. You'll never need most of the lines in the file.

For getting summary data, just keep running totals on the data you care about. For min and max values that's easy, but you can even keep running averages and such fairly easily. 

my ($avg, $num_contrib) = (0, 0);
foreach (<get a line>) {
    my $val = <something>;
    $avg += ($val - $avg) / ++$num_contrib;
}

[download]
Other kinds of running totals can be gotten by similar algorithms.

Naturally, you'd want to combine all these things so you're only doing a single pass through each file, for efficiency.

Ron Steinke
<rsteinke@w-link.net>

In reply to Re: Design Approach (Processing Huge Hash) by rsteinke
in thread Design Approach (Processing Huge Hash) by mkirank

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.