You should see if you can get some kind of more sophisticated indexing system. I don't remember if Glimpse speeds up within-file sorts, but if it does you could use it with "agrep". (Google(TM) it).

I haven't worked with the module Search::InvertedIndex, but you could still use it, or a similar approach. You need to keep a list of all the indexed words so that you can do a fast serial scan over it (I don't know if Search::InvertedIndex will allow this) and see which of these your pattern matches. Then you look those up in the InvertedIndex to get the list of actual matches. You should probably do a merge/sort of all the matches before you retrieve them from the actual data file.

In reply to Re: speeding up a file-based text search by Thelonius
in thread speeding up a file-based text search by perrin

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.