Perl-Sensitive Sunglasses | |
PerlMonks |
Re^4: Index a file with pack for fast accessby BrowserUk (Patriarch) |
on Dec 21, 2011 at 16:13 UTC ( [id://944655]=note: print w/replies, xml ) | Need Help?? |
As an aside, one characteristic of SQLite that “bit me bad” at first is the way that this system handles transactions. Basically, you must have one, because if you don’t, SQLite will physically verify every single disk write by reading the information again. Which certainly can result in the “hours or days” concern, and then rather dramatically relieve that concern. Transactions aren't involved when using SQLite's bulk loader. The syntax is simply:
But if you do that alone on a csv file containing 16 million records, you'll wait days. Try it for yourself. And doing it via SQL inserts, even with transactions, will take even longer. Again, try it for yourself. However, if you precede the .import with the appropriate bunch of seven PRAGMA commands, then the entire import takes just over 2 minutes. But finding/working out/remembering those 7 pragmas is non-trivial. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
In Section
Seekers of Perl Wisdom
|
|