perrin kindly provided a link, and the best thing to do would be to get the module and read the docs. You're trying to index a flat-file database, and its just not scaling well. Search::InvertedIndex uses either MySQL or DB_File on the backend, and if you don't want to use MySQL, you should get DB_File. An 'inverted index' search just happens to be exactly what you're doing. Your main database is indexed by some id value, and its easy to look things up by the index, but you don't want to look things up by that id value, you want to look up those id values by some keywords. That's what Search::InvertedIndex does, and since the work has been done for you, you're advised to take advantage of it (and you can always look at the code if you're curious as to how its implemented).

In reply to Re: Re: Re: All-Perl search engine having speed issues by runrig
in thread All-Perl search engine having speed issues by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.