in reply to Hitting memory limit? (ActiveState Perl)

It's possible that you're encountering the practical limits of using flat files and in-memory hashes. Perhaps its time for a database approach. With a proper (and simple) database and some redesign of how you're doing things, you would never need to hold all users in memory at the same time, and wouldn't have to worry about running out of memory, even as the user list grows.

Without seeing actual code, it's difficult to give any more specific tips, but in general, when you start running out of memory it's time to reconsider your design philosophy. Often this means taking that step into the world of databases.

The DBI module, and DBD::SQLite can help to ease the pain of such a transition.

Update: Corrected typo where DBD::SQLite was referred to as DBM::SQLite


Dave

  • Comment on Re: Hitting memory limit? (ActiveState Perl)

Replies are listed 'Best First'.
Re: Re: Hitting memory limit? (ActiveState Perl)
by NYTAllergy (Initiate) on Jan 22, 2004 at 14:14 UTC
    In terms of moving to a database, but still staying within Perl, are the aforementioned DBI and DBM:SQLite the "best" options? Are there others that I should look at as well, to make sure that I'm using the interface / dbase that will work best for my particular use?
[reply]
      A database solution allows for fast "random" read/write access to data. This can assist you in overcoming the need to build large in-memory hashes. This may be only one solution, but can be a powerful and highly expandable one.

      Your logfiles could even be written to databases, though that's probably less important than being careful to not try to slurp too much in at a time from your logfiles as you parse them. The key is to parse, and write out to the database without building up gigantic hashes of usernames and other data.

      To this end, DBI and DBD::SQLite are very good options. They will require that you learn some SQL. But DBD::SQLite is such a powerful (albiet lightweight) database that you are likely to not need anything heavier (such as Oracle) for a long time. DBD::SQLite is the database driver, and the database server wrapped into one module. DBI.pm is the "Database Interface" which allows you to write code in a database independant way, rather than getting down to the nitty-gritty of dealing directly with a low level database driver. You use the two together: DBI as the API to DBD::SQLite. I do consider them to be very good options if you're looking for ways to scale your project beyond its current capacity.


      Dave

[reply]

In Section Seekers of Perl Wisdom