in reply to Re: DBIx::Class and Parallel SQLite Transactions
in thread DBIx::Class and Parallel SQLite Transactions

Corion, thanks for your response.

One of the reasons that I selected DBIx::Class as an ORM was that it detects threads/forks and manages the DBI handles approrpriately. I really don't think I want to get into writing and managing code to do this on my own.

I am thinking hard about my continued use of SQLite. Given what I am doing, there is benefit in using a file based DB. It give me flexibility in moving the execution from machine to machine without having to manage a database move/replication. However, it does come with many limitations, concurrency and write performance being the main ones.

This query is probably my last stop on the continued use of SQLite for this project. If I don't get any eye opening responses on how to accomplish my goals with SQLite, I will be moving to another DB, probably MySQL since I have other applications that use it.

Thanks again,

lbe

  • Comment on Re^2: DBIx::Class and Parallel SQLite Transactions

Replies are listed 'Best First'.
Re^3: DBIx::Class and Parallel SQLite Transactions
by Corion (Patriarch) on Jul 14, 2011 at 12:58 UTC

    DBIx::Class is easily fork/threadsafe, as it is pure Perl. But its README says

    ... [ DBIx::Class ] is fork- and thread-safe out of the box (although your DBD may not be).

    ... and I don't expect DBD::SQLite to be fork and/or threadsafe. Likely, the SQLite documentation and/or DBD::SQLite documentation even say so.

      To the best of my knowledge, DBIx::Class is fork-safe with all DBD modules, because it just opens a new connection if it detects that a fork has happened (by comparing the current PID to the one that was recorded at the time of the last DB connect, or something similar).

      I'm certainly no expert here. My belief that DBIx::Class with SQLite is threadsafe is based upon not just the CPAN documentation that you referenced but also several threads that I have read from the DBIx::Class developers. This thread from GrokBase seems to be the most complete, at least that I remember, : [Dbix-class ] "fork- and thread-safe"

      Things certainly seem to behave properly when reading from SQLite; however, I can't say the same for writing to SQLite.

      But as I previously mentioned, this appears to be an issue with how the lock state of the database is handled. I haven't tested this across separate autonomous processes, though while typing this note I thought of a way I can probably test this without too much work. I'll give it a try this evening when I have a little time.

        But as I previously mentioned, this appears to be an issue with how the lock state of the database is handled.

        The write performance of SQLite will be far less than say MySQL. SQLite needs to gain an exclusive lock on the entire DB file in order to do a write. There is no table level or row level locking. So I figure that having multiple writers will bring nothing but trouble to you. Since in a transaction your rows are exclusive (don't overlap with other transactions), then a DB that can do row level locking could yield a lot higher performance.

        I am working on a SQLite project and have found the O'Reilly book, "Using SQLite" by Jay A. Kreibich to be helpful. It is mainly oriented around the C interface, but there is plenty of great info for the Perl users too. There are discussions about how and why the DB can be busy - it is locking related.

        Update: Oh, I'm not sure how well the DBI does with threads. The implementation may be "safe", but not high performance, i.e. it may just wind up serializing things. With MySQL, you may get higher performance with a process per writer instead of a thread. Benchmarks will tell.

        I see no mention of threads and forking in the DBD::SQLite documentation. Grepping the source also lists no mention of threads in the SQLite.xs file, but only within sqlite3.c itself. This indicates to me that little thought has been given to how to make DBD::SQLite and threads/fork play nice together.

        You can maybe check whether the problem is inherent to fork() and/or threads by launching multiple writes as real, separate processes via exec resp. system instead of fork/threads. If you still experience the segfaults/crashes, the problem likely is with your version of DBD::SQLite or some other XS code loaded in the separate process. If the problem goes away, then the problem likely is related to fork/threads and I see no other way than to change to a different DBD if you want to keep going with fork/threads.

        You haven't told us your OS so far, but if you are on Windows, threads and fork() are basically the same thing there anyway, and all problems with one system are present in the other as well.