Re: DBI::SQLite slowness (DBD::SQLite bulk insert woes)

.echo ON

.read create_table_without_pk.sql

PRAGMA cache_size = 400000;
PRAGMA synchronous = OFF;
PRAGMA journal_mode = OFF;
PRAGMA locking_mode = EXCLUSIVE;
PRAGMA count_changes = OFF;
PRAGMA temp_store = MEMORY;
PRAGMA auto_vacuum = NONE;

.separator "\t"
.import a_tab_seprated_table.txt mytable

BEGIN;
.read add_indexes.sql
COMMIT;

.exit

sqlite3 mydb.db < commands.txt

[download]

Replies are listed 'Best First'.
Re^2: DBI::SQLite slowness (DBD::SQLite bulk insert woes) by Anonymous Monk on Sep 20, 2013 at 02:45 UTC
See also SQLite Optimization FAQ (old but informative) and SQLite Slow insert (use transactions)	[reply]
Re^3: DBI::SQLite slowness (DBD::SQLite bulk insert woes) by marinersk (Priest) on Sep 20, 2013 at 20:31 UTC
I can attest to efficiency improvements due to at least one of the notes in SQLite Optimization FAQ. Used it at a customer site once, too long ago to remember the numbers but I recall the difference was dramatic. Sample of the commands needed: C:\Steve>sqlite test.db C:\Steve>call sqlite-STEVEM2 test.db C:\Steve>"C:\App\SQLite\sqlite3.exe" test.db SQLite version 3.3.5 Enter ".help" for instructions sqlite> pragma default_cache_size; 2000 sqlite> pragma default_cache_size=200000; sqlite> pragma default_cache_size; 200000 sqlite> .exit C:\Steve>sqlite test.db C:\Steve>call sqlite-STEVEM2 test.db C:\Steve>"C:\App\SQLite\sqlite3.exe" test.db SQLite version 3.3.5 Enter ".help" for instructions sqlite> pragma default_cache_size; 200000 sqlite> .quit C:\Steve> [download]	[reply] [d/l]