Re: Concurrent access with BerkeleyDB

Once you have locking initialized in your BerkeleyDB environment (which it sounds like you do), you can mostly forget about it. BerkeleyDB will automatically acquire and release locks as needed. You only need to give the DB_WRITECURSOR flag if you really want to use a cursor to write to the database. If you do all your writes with db_put or tied hash assignments, you don't need to worry about it.

Comment on Re: Concurrent access with BerkeleyDB

Replies are listed 'Best First'.
Re: Re: Concurrent access with BerkeleyDB by knowmad (Monk) on Feb 20, 2004 at 09:44 UTC
Thanks for the feedback. That's what I've read but it's not what I'm seeing in my tests which fork two children and try to do concurrent inserts. I can watch the database flying through the inserts and then eventually get hung and never (well at least within 15 minutes ;-) release. I even went to the trouble of enabling transactions but that did not prevent the lock from occurring. I have had some initial success with limiting the input via `sleep 1` (too slow for effective testing) or `select (undef,undef,undef,0.25)` as described in the perlfaq8. This "solution" seems more like a hack for an inefficient locking system. Is this a limitation of BerkeleyDB.pm or in the BSDDB libraries? Or perhaps it's my poor understanding of forking which is at fault. My concern in getting this right is that I have a client with a large database of records which I'd hate to corrupt (I'm weaning them away from Python and corrupting the database would not be a good thing for proving Perl's effectiveness). I've seen several posts by MoveableType users who have had corruptions with BDB. Most of the data is recoverable but the downtime is both annoying and expensive (when it's a commercial business). Yes, I am considering moving the client to stronger database solution such as PostgreSQL or MySQL. I just wanted to fully exhaust this line of reasoning before recommending the switch. Sleepycat has done some nice work with BerkeleyDB but if it can't do reasonable concurrency then I won't risk using it in a production environment. I'll post my test script in Meditations in the hope that someone can point out the flaw in my logic or in my implementation of BerkeleyDB. Wm	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re: Re: Concurrent access with BerkeleyDB
by knowmad (Monk) on Feb 20, 2004 at 09:44 UTC

Thanks for the feedback. That's what I've read but it's not what I'm seeing in my tests which fork two children and try to do concurrent inserts. I can watch the database flying through the inserts and then eventually get hung and never (well at least within 15 minutes ;-) release.

I even went to the trouble of enabling transactions but that did not prevent the lock from occurring. I have had some initial success with limiting the input via sleep 1 (too slow for effective testing) or select (undef,undef,undef,0.25) as described in the perlfaq8.

This "solution" seems more like a hack for an inefficient locking system. Is this a limitation of BerkeleyDB.pm or in the BSDDB libraries? Or perhaps it's my poor understanding of forking which is at fault.

My concern in getting this right is that I have a client with a large database of records which I'd hate to corrupt (I'm weaning them away from Python and corrupting the database would not be a good thing for proving Perl's effectiveness). I've seen several posts by MoveableType users who have had corruptions with BDB. Most of the data is recoverable but the downtime is both annoying and expensive (when it's a commercial business).

Yes, I am considering moving the client to stronger database solution such as PostgreSQL or MySQL. I just wanted to fully exhaust this line of reasoning before recommending the switch. Sleepycat has done some nice work with BerkeleyDB but if it can't do reasonable concurrency then I won't risk using it in a production environment.

I'll post my test script in Meditations in the hope that someone can point out the flaw in my logic or in my implementation of BerkeleyDB.

[reply]
[d/l]
[select]