Re^3: A per-instance shared queue among forked child processes?

I would be very reluctant to use SQLite in such a situation, out of fear that “this is not really the sort of thing that its designers (probably) had in mind.” I am categorically nervous about going too far outside the envelope.

If you have “4,000 inserts to do in 2.5 seconds,” I wonder if you could get away with somehow batching the queue-entries. If there is any sort of queue-buildup at all, maybe you could hold ’em until you’ve got, say, 10 entries accumulated and then drop them into the queue all at once. (They would be dequeued and processed in similar groups.) This would reduce queue overhead by 10% and maybe those microseconds are otherwise adding-up to a useful-to-avoid amount of time. Sometimes they do ...

Replies are listed 'Best First'.
Re^4: A per-instance shared queue among forked child processes? by BrowserUk (Patriarch) on Apr 05, 2011 at 19:22 UTC
I would be very reluctant to use SQLite in such a situation, out of fear that “this is not really the sort of thing that its designers (probably) had in mind.” I am categorically nervous about going too far outside the envelope. I'd love to see the basis of your fear. Is it founded in some documentation you've read? Or some demonstrably practical limitation you've encountered? My question above was based upon practical experience of SQLite 2 where writer starvation was a frequent limitation in high concurrency use. salva's demonstration of 800/s throughput conclusively scotches that as a problem with SQLite 3. And further research leads to specific documentation of both the original problem and the solutions implemented in SQLite 3 to address it. In the absence of relevant hands-on experience, gut feel is a notoriously bad indicator of anything. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]
Re^4: A per-instance shared queue among forked child processes? by salva (Canon) on Apr 06, 2011 at 07:53 UTC
out of fear that “this is not really the sort of thing that its designers (probably) had in mind.” All I see are SQL `select`, `insert`, `update`, `delete`... what do you fear? what do you thing the designers had in mind?	[reply] [d/l] [select]