Problem

multiple servers write to a log file the log file will have entries for several days for each day for each server summarize the data and print the top x elements

Logfile Format

servername,epoch time,Field1,Feild2,Field3,Field4 (since multiple servers are writing the epoch time is not in an order)
Below is the flow of the intended program (there may be some syntax errors)
1.The Logfile has around 5-6 million entries

2.Is their a better approach (like summarizing the data as we are reading it from the logfile , since it will take a lot of memory .
3. for summarization again we need to create a hash and take the top x elements of that hash 



use Heap;

my %servers;
my $servername;

my ($epoch,$field1,$field2,$field3,$field4);

my $tmparref;
my $heap = 'heap';
open (LOG,"$file") or die "cannot open $!";
while (<LOG>) {
    ($servername,$epoch,$field1,$field2,$field3,$field4) = get_details
+($_);
    ## Create a reference to a array
    $tmparref = [$field1,$field2,$field3,$field4]

    if (exists($servers{$servername}{$epoch})) {
               push@{$servers{$servername}{$epoch}},$tmparref;
    } else {
               $servers{$servername}{$epoch}=[$tmparref];
           ## creat a object of heap dynamically at runtime

           ${$heap$servers{$servername}} = Heap->new;

    }
    ## for each server add the epoch time to a heap object
    ${$heap$servers{$servername}}->add( $elem );

    
}

close(LOG);
my ($process_server,$minday,$maxday);
foreach (keys (%servers) {
     $process_server = $servers{$_};
     ##
      Sort on the keys (epoch)
      get the minimum and maximum epoch for that server

      for (then for each day) {
          ### Here again we need to create a hash with the summarized 
+data
      ###
      summarize the data
  
      }
        print the record for top x (a constant)

}

[download]

In reply to Design Approach (Processing Huge Hash) by mkirank

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.