Hello Gtforce,

without other details (it's the data already in memory?) it's a matter of guessing, but..

pass by reference? Are you filling new datastructures to pass your data around? If the original data it's alreaady read you can take a reference to some items and pass just the reference to your subroutines. See also the same matter at effective perl programmer and Is it possible to do pass by reference in Perl?

This big amount of data and relations between parts let me think of databases: you can have a first step where you put all your data into a (temporary?) database (sqlite for example) and a second step where you just SELECT appropriate items and you update your statistics.

Be sure of freeing no more used variables. Four days for just 2 millions records seems a bit slow for me.

If my suggestions do not fit your task please provide a bit more details.

HtH

L*

There are no rules, there are no thumbs..
Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.

In reply to Re: creating and managing many hashes by Discipulus
in thread creating and managing many hashes by Gtforce

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.