Re: Implementing a buffered read-and-insert algorithm

I would check if your database supports delayed inserts. That will allow you to do your buffered read and the a sort of bufferd insert. I'm not sure what that will do as far as disk reads and writes, but i've had great success with this startegy in Database->Database transfers where the insert time was holding me up. (By great success I mean from 60+seconds per table to 5-10 seconds per table)

___________
Eric Hodges

Comment on Re: Implementing a buffered read-and-insert algorithm

Replies are listed 'Best First'.
Re^2: Implementing a buffered read-and-insert algorithm by radiantmatrix (Parson) on Dec 13, 2004 at 21:58 UTC
According to the MySQL manual, `INSERT DELAYED` gets its benefit from allowing chunks of data from many different clients to be inserted as one block. So, even if I were using MySQL, this particular application would not benefit from it. By reading chunks of data and then inserting them in rapid sequence, I have managed to ask the database to write in larger blocks -- and that has increased performance. But, you solved another bugger of a problem I've been having, saving me another SoPW post. Thank you! ;-) radiantmatrix `require General::Disclaimer; s//2fde04abe76c036c9074586c1/; while(m/(.)/g){print substr(' ,JPacehklnorstu',hex($1),1)}`	[reply]
Re^3: Implementing a buffered read-and-insert algorithm by eric256 (Parson) on Dec 13, 2004 at 22:09 UTC
Glad to have helped in someway. The previous paragraph in the documentation shows the advantage in your situation. Like i said before with them both on the same machine this might not yeild a great amount of benifit, but when transfering between machines its great. When a client uses INSERT DELAYED, it gets an okay from the server at once, and the row is queued to be inserted when the table is not in use by any other thread. Another major benefit of using INSERT DELAYED is that inserts from many clients are bundled together and written in one block. This is much faster than doing many separate inserts. ___________ Eric Hodges	[reply]