Have you considered using benchmark to time your results?

I reckon there is a better technique of parsing your elements than loading them all into memory, processing, then collecting the garbage.

Where is you data coming from?

  • If its dbi, you could do some funky chicken with selects and loops.
  • If its a ff, you could parse one line at a time, process it, and stick it on a stack to be whisked off to your db. (Not sure if the overhead of many I/O's would take more time than processing 100,000,000 rows in memory)
  • If your data is sparse, perhaps you could do some "preprocessing" on the source before the "real" processing begins.
  • What about fork or more scripts if you have multiple datasources?
  • If you are collecting your garbage just before your script ends, you may not need to as perl will free the memory when it ends!
  • Are you collecting the garbage to increase the performance of your DBI calls? (ie pages to disk) - is there a trade off to be considered here?

    Dont know if this will help but as they say (and I am more than aware of) - "You cant think of everything all the time."

    HTH

    In reply to Re: Helping the garbage collector by Ryszard
    in thread Helping the garbage collector by vlakkies

    Title:
    Use:  <p> text here (a paragraph) </p>
    and:  <code> code here </code>
    to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.