in reply to Re: Hitting memory limit? (ActiveState Perl)
in thread Hitting memory limit? (ActiveState Perl)

In terms of moving to a database, but still staying within Perl, are the aforementioned DBI and DBM:SQLite the "best" options? Are there others that I should look at as well, to make sure that I'm using the interface / dbase that will work best for my particular use?
  • Comment on Re: Re: Hitting memory limit? (ActiveState Perl)

Replies are listed 'Best First'.
Re: Re: Re: Hitting memory limit? (ActiveState Perl)
by davido (Cardinal) on Jan 22, 2004 at 14:22 UTC
    A database solution allows for fast "random" read/write access to data. This can assist you in overcoming the need to build large in-memory hashes. This may be only one solution, but can be a powerful and highly expandable one.

    Your logfiles could even be written to databases, though that's probably less important than being careful to not try to slurp too much in at a time from your logfiles as you parse them. The key is to parse, and write out to the database without building up gigantic hashes of usernames and other data.

    To this end, DBI and DBD::SQLite are very good options. They will require that you learn some SQL. But DBD::SQLite is such a powerful (albiet lightweight) database that you are likely to not need anything heavier (such as Oracle) for a long time. DBD::SQLite is the database driver, and the database server wrapped into one module. DBI.pm is the "Database Interface" which allows you to write code in a database independant way, rather than getting down to the nitty-gritty of dealing directly with a low level database driver. You use the two together: DBI as the API to DBD::SQLite. I do consider them to be very good options if you're looking for ways to scale your project beyond its current capacity.


    Dave