#!/usr/bin/perl

use strict;
use warnings;

use POSIX qw(_exit);
use DBI;
use File::Temp;

$| = 1;
my $dbfile = "/tmp/db-$$";

my $dbh = DBI->connect("dbi:SQLite:$dbfile", "", "", {RaiseError => 1}
+);

$dbh->do("create table queue (child, ix)");

for my $id (0..3) {
    fork or do {
        print "<$id enter>";
        my $dbh = DBI->connect("dbi:SQLite:$dbfile", "", "", {RaiseErr
+or => 1});
        $dbh->do("PRAGMA journal_mode = OFF");
        my $sth;
        while (not defined $sth) {
            $sth = eval { $dbh->prepare("insert into queue values (?, 
+?)") };
        }
        print "<$id with sth>";
        for (1..1000) {
            eval {
                $sth->execute($id, $_);
                print $id;
            };
            $@ and print "<$id error: $@>"
        };
        print "<$id exit>";
        _exit(0);
    }
}

1 while (wait > 0);

my $sth = $dbh->prepare("select count(*) from queue");
$sth->execute;
my $row = $sth->fetchrow_arrayref;
print "\n@$row\n";
[download]

update: It seems that even better than disabling the journal, it is to disable synchronous writes (= ensuring that written data actually gets into the disk surface):

$dbh->do("PRAGMA synchronous = OFF");
[download]

That makes 4000 inserts in 2.5 seconds in my computer, and is safer as the database would not get corrupted unless the OS failed to flush cache data to disk.

A more complete simulation:
Read more... (5 kB)

I would be very reluctant to use SQLite in such a situation, out of fear that “this is not really the sort of thing that its designers (probably) had in mind.”   I am categorically nervous about going too far outside the envelope.

If you have “4,000 inserts to do in 2.5 seconds,” I wonder if you could get away with somehow batching the queue-entries.   If there is any sort of queue-buildup at all, maybe you could hold ’em until you’ve got, say, 10 entries accumulated and then drop them into the queue all at once.   (They would be dequeued and processed in similar groups.)   This would reduce queue overhead by 10% and maybe those microseconds are otherwise adding-up to a useful-to-avoid amount of time.   Sometimes they do ...

I would be very reluctant to use SQLite in such a situation, out of fear that “this is not really the sort of thing that its designers (probably) had in mind.” I am categorically nervous about going too far outside the envelope.

I'd love to see the basis of your fear. Is it founded in some documentation you've read? Or some demonstrably practical limitation you've encountered?

My question above was based upon practical experience of SQLite 2 where writer starvation was a frequent limitation in high concurrency use. salva's demonstration of 800/s throughput conclusively scotches that as a problem with SQLite 3. And further research leads to specific documentation of both the original problem and the solutions implemented in SQLite 3 to address it.

In the absence of relevant hands-on experience, gut feel is a notoriously bad indicator of anything.

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

out of fear that “this is not really the sort of thing that its designers (probably) had in mind.”

All I see are SQL select, insert, update, delete... what do you fear? what do you thing the designers had in mind?