glwtta has asked for the wisdom of the Perl Monks concerning the following question:

Hi everyone,

I've introduced a Cache::BerkeleyDB based caching mechanism into a web application I'm working on, and it seems to work fine at first, but breaks down when I give it a bit more data.

I have an expensive computation that pre-loads the cache with about 20,000 values. Running it reproduceably wedges the bdb database:

Can't create BerkeleyDB::Env (home=/data/webdata/cache): DB_RUNRECOVERY: Fatal error, run database recovery at /apps/perl-5.8.6/lib/site_perl/5.8.6/Cache/BerkeleyDB_Backend.pm line 29, <DATA> line 225.

Running the exact same query with a smaller input works fine. This is under mod_perl, but I have been testing it with a single httpd process, so I don't think it's a concurrency issue.

Any idea where I should look to solve this?

In the meantime I've been using Cache::File instead, but both the performance and space-efficiency are unacceptable (I think it would be ok on something like reiserfs, but I'm stuck on ext3). Because of the size of the data memory based caches won't work for me, so BDB was really the all-around best solution; if I can't get this working I'll have to go to a relational database, and I'd rather avoid the hassle.

Thanks in advance.

Replies are listed 'Best First'.
Re: Cache::BerkeleyDB Problems
by kvale (Monsignor) on May 10, 2006 at 03:55 UTC
    It may be that you are running out of cache. Try hacking the BDB env setup: 
    my $env = new BerkeleyDB::Env
-Cachesize => 131072,  # or larger
...

    [download]

    -Mark

[reply]
[d/l]
      Hmm, ironic. I'll take a look at this, though I'm not quite sure how to make Cache::BerkeleyDB do this.
[reply]
      That was it! I'm going to contact the module author about passing more configuration options to BDB Env.

      What do I have to keep an eye on that concerns the cache usage? Is it the number of keys? Largest entry? Total size?

      Thanks for the help.

[reply]
Re: Cache::BerkeleyDB Problems
by PodMaster (Abbot) on May 10, 2006 at 18:08 UTC
    BerkeleyDB DB_RUNRECOVERY =>
      Berkeley DB Reference Guide: Error returns to applications

    DB_RUNRECOVERY

    There exists a class of errors that Berkeley DB considers fatal to an entire Berkeley DB environment. An example of this type of error is a corrupted database page. The only way to recover from these failures is to have all threads of control exit the Berkeley DB environment, run recovery of the environment, and re-enter Berkeley DB. (It is not strictly necessary that the processes exit, although that is the only way to recover system resources, such as file descriptors and memory, allocated by Berkeley DB.)

     The log files ought to be able to tell you more about the cause of the error. Recovery procedures can help you recover your database.

    MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
    I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
    ** The third rule of perl club is a statement of fact: pod is sexy.

[reply]
      Yes, I understand how to recover the database, I'm trying to make it not break in the first place :)

      Logging doesn't seem to be turned on, any idea how I can do that with Cache::BerkeleyDB?

      Thanks!

[reply]
Re: Cache::BerkeleyDB Problems
by perrin (Chancellor) on May 10, 2006 at 20:31 UTC
    If you can't solve the issue, you may want to look at Cache::FastMmap instead.
[reply]

Back to Seekers of Perl Wisdom