in reply to fast disk db with bulk insert, fast read access, compact storage

Maybe you should tell us a little more about your actual application before we get too distracted by "I can't do X because Y" without having any justification for the assertion? Interesting additional information may be:

  1. How many read accesses per (whatever appropriate period of time)?
  2. Will the same 'word' be accessed multiple times per period?
  3. Is the same root comprising the first n characters likely to be accessed multiple times per period
  4. What is an appropriate period?
  5. What are the constraints on the time it takes to build the 'database'?
  6. What is 'very fast'?
  7. Is there any point compressing the values?
  8. What else might influence a smart solution?
True laziness is hard work
  • Comment on Re: fast disk db with bulk insert, fast read access, compact storage

Replies are listed 'Best First'.
Re^2: fast disk db with bulk insert, fast read access, compact storage
by iaw4 (Monk) on Sep 18, 2010 at 07:25 UTC

    thanks everyone for the pointed questions. I had tried to keep my question reasonably general, because I thought it would be better for others who have similar questions.

    the specific application is a data base of published articles. think of

      unique-key|Time Magazine|Why the monks are great|Sep 13, 2006|p245-133|volume 8|number 10

    the data base, in plain text and this form, is about 5GB now (but could grow to 20GB in the future), and so the ASCII version fits into RAM. usually the DB changes, say, once per month. I could rebuild it every time anew from the store. there is no guarantee on length or uniqueness of anything, except the unique key.

    I do need quick access into individual words. so, if I want to find all articles that contain the word 'Time' and the work 'monks' and the number 245, my search should be blindingly fast to find all unique-keys that contain the three words, and then display these records. assume access is very frequent, too---say, I wanted to do research that does 'permutation of words' research, so each article launches a search over the data base.

    the lazy implementation would be to take every word, and put each word as key into a hash with the value being the arrays of unique keys where the word occurs; and a second hash which gives me the record given a unique key. of course, with perl hashes, this would take too much space. from my limited experience with SQL, after I rearrange the data, it would also blow up a lot.

    on the plus side, this is all "read-only".

    sql would be ok, but it just feels like it is not the right tool for the job. sql dbs seem made more for updating than for blindingly fast read access.

    I was also only guessing that SSD would be a good tool for the job.

    help?

[reply]
[d/l]

      Your original description of your application:

      simple---key, value. ... 32GB ... (think as application of a word hash that I am rebuilding every night, and I want to do real-time search as my users are typing words.)

      Is almost completely at odds with this description:

      is about 5GB now (but could grow to 20GB in the future) ... the DB changes, say, once per month ... I do need quick access into individual words. so, if I want to find all articles that contain the word 'Time' and the work 'monks' and the number 245, my search should be blindingly fast to find all unique-keys that contain the three words, and then display these records.

      The former implies indexing by the characters of the unique key only.

      The latter requires a fully inverted index of the words in the entire records, which essentially makes the unique key redundant.

      You need to define the actual use your data will be put to, before looking for the mechanism for doing it.

[reply]

        actually, I think the db descriptions are pretty much the same (5GB for testing now, 20GB in the future, so 32GB is a good upper limit), although neither description was very good. however, what is very good is that you told me what it is that I am really looking for: an inverted index. thanks a lot. very helpful. I should be able to search for this now in a much more intelligent fashion.

        so I need a nice, fast inverted index program for ubuntu perl. knowing what I need, I could now search cpan. as luck would have it, Search::Moose seems to be designed for this sort of job. (as bad luck would have it, it aborts during the build stage on my ubuntu machine. if someone knows more about Search::Moose, please let me know.)

        thanks a lot, everybody.

        regards, /iaw

[reply]

In Section Seekers of Perl Wisdom