A database solution allows for fast "random" read/write access to data. This can assist you in overcoming the need to build large in-memory hashes. This may be only one solution, but can be a powerful and highly expandable one.

Your logfiles could even be written to databases, though that's probably less important than being careful to not try to slurp too much in at a time from your logfiles as you parse them. The key is to parse, and write out to the database without building up gigantic hashes of usernames and other data.

To this end, DBI and DBD::SQLite are very good options. They will require that you learn some SQL. But DBD::SQLite is such a powerful (albiet lightweight) database that you are likely to not need anything heavier (such as Oracle) for a long time. DBD::SQLite is the database driver, and the database server wrapped into one module. DBI.pm is the "Database Interface" which allows you to write code in a database independant way, rather than getting down to the nitty-gritty of dealing directly with a low level database driver. You use the two together: DBI as the API to DBD::SQLite. I do consider them to be very good options if you're looking for ways to scale your project beyond its current capacity.


Dave

In reply to Re: Re: Re: Hitting memory limit? (ActiveState Perl) by davido
in thread Hitting memory limit? (ActiveState Perl) by NYTAllergy

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.