In case anyone hasn't guessed.. I'm very interested in this thread because I am running into similar problems with some code that I'm writing. I'm not running out of memory but I am noticing that the Perl strucutres I am storing things in are getting very large and unwieldy and RSS when tracked is getting HUGE. Gee... I wonder just how many Perl Monks and non-PM Perl coders are getting sucked into virus remediation right now.

All that aside: I have thought about using DBMs and/or an RDBMS to store intermediate results for my scripts as well. The problem with that is you tend to lose performance using that approach (especially using DBI) due to the overhead of communicating with the database and the scripts I'm writing are too slow already.

Of course, as the OP pointed out indirectly the problem with keeping everything in memory is you can concievably run out of memory when your data structures get REALLY huge due to the amount of data you are processing.

This approach works OK too if your data "cooperates" and lines itself up really nice for you the way you want it. However when you are processing logs you may have related events that you want to track that are many many lines apart in the log files.

I know I am not coming up with a solution here, but I can feel smellysocks's pain here as I'm trying to solve some of the same problems and I'm not coming up with any good answers either.

Peter L. Berghold -- Unix Professional
Peter at Berghold dot Net
Chat Stuff: AIM:  redcowdawg
Yahoo: blue_cowdawg
Cowdawg Philosophy:  Sieze the Cow! Bite the Day!
Clever Quip:  Nobody expects the Perl Inquisition!
Non-Perl Passion:   Dog trainer, dog agility exhibitor, brewer of fine Belgian style ales. Happiness is a warm, tired, contented dog curled up at your side and a good Belgian ale in your chalice.

In reply to RDBMS vs. RAM (was: Remaining Memory on a system) by blue_cowdawg
in thread Remaining Memory on a system by smellysocks

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.