I'm trying to build a rather large read-only database with millions of keys. The MySQL/Postgres/etc. route is not available to me; I need to do this entirely through Perl.
So I decided on GDBM_File for this.
Now, it will let me do
tie %hash, 'GDBM_File', $filename, &GDBM_WRCREAT, 0640;
The problem is, the keys are sorted and the script gets bogged down rearranging data on disk as it is being built (from looking at the output of
top, it is perpetually on disk wait. it doesn't help that the disk is a slow IDE drive :( ). My question is: is it possible to build the database in memory, and then just write it out in one fell swoop?
Update: A simple little flag made a _WORLD_ of difference. We're talking a 1000x increase in performance here. I added &GDBM_FAST to the tie flags, and this thing flies now:
tie %hash, 'GDBM_File', $filename, &GDBM_WRCREAT|&GDBM_FAST, 0640;
Thank you everyone for replying; even if your replies didn't solve the problem, I am grateful that you took the time to respond. :-)
Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
Read Where should I post X? if you're not absolutely sure you're posting in the right place.
Please read these before you post! —
Posts may use any of the Perl Monks Approved HTML tags:
- a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
| |
For: |
|
Use: |
| & | | & |
| < | | < |
| > | | > |
| [ | | [ |
| ] | | ] |
Link using PerlMonks shortcuts! What shortcuts can I use for linking?
See Writeup Formatting Tips and other pages linked from there for more info.