we do plan to use a database but the problem is that the script has to support different databases (postgres,oracle,Ms-sql).
If we use a database , then inserting all the records will take much more time (we cannot do a bulk insert as we deal with different databases) ,so we came up with the idea of summarizing as this will reduce the number of inserts to the database
As per your suggestion , what i can probably do is insert all the records into SQLLite (as this insertion will be faster than other databases) and then probably summarize the data through SQLlite and then insert that summarized data into postgres or other databases.
Thanks for your comments .

In reply to Re^2: Design Approach (Processing Huge Hash) by mkirank
in thread Design Approach (Processing Huge Hash) by mkirank

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.