Re: DBI::SQLite slowness

When a hash based solution starts to choke because of the amount of data, the easiest solution is to use an on-disk hash. See BerkeleyDB or DB_File. The only change necessary will be to load the module and tie the hash. The rest of the script will stay the same.

Jenda
Enoch was right!
Enjoy the last years of Rome.

Comment on Re: DBI::SQLite slowness

Replies are listed 'Best First'.
Re^2: DBI::SQLite slowness by Laurent_R (Canon) on Sep 21, 2013 at 09:24 UTC
I tried that a couple of time, the performance on inserting new records just became really bad. It may be OK when the DB file already exists and is more or less static, but apparently not when you are constantly adding new records. It may also be that I did not do it the right way, but I tried various things, I don't see what I might have done wrong.	[reply]
Re^3: DBI::SQLite slowness by Jenda (Abbot) on Sep 21, 2013 at 16:10 UTC
It will always be quite a bit slower than when the hash fits in memory, but still much quicker than if the hash grows so big that the computer has to swap memory pages onto disk. Jenda Enoch was right! Enjoy the last years of Rome.	[reply]
Re^4: DBI::SQLite slowness by Laurent_R (Canon) on Sep 21, 2013 at 16:48 UTC
Agreed.	[reply]