Hi, to be frank it seems like you need a real search engine. This is covered in some detail in the book CGI programming with Perl from O'Reilly aka the rat book. I recommend this to you.

From the code you have posted it seems you are searching a flat text file form begining to end which as you note does not scale well - ie it gets slower and slower the bigger it gets. The basis of a search is to either use a real database or generate an index that you can search (usually via a hash key). You do the processing in advance when you generate this index and then your CGI searches the index to find what you want. You try to limit the processing that needs to be done in real time (ie for the CGI) so things happen fast from the user's point of view.

A good free search engine you can incorporate into your site is available from http://www.whatuseek.com/ I use this for cheap and cheerful searches on shoestring sites. You can customise the results page into a format that looks like your site. Downside is ads and a page limit. You can see an example of this in action here It is not as good as your own Perl engine could be but is fast and easy to set up. View the source to see how the search box links to the search engine.

Good luck. If you post more code or probably links to what you currently have we may be able to suggest how to speed it up for you. It is not really clear what you want to search for.

Update

Forget the database. I have written a little search app for you that will grep out all the lines that match a given search criterion in your data file in ~16 milliseconds. See below. This should be fast enough for you :-)

cheers

tachyon

s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

In reply to Re: Faster Search Engine by tachyon
in thread Faster Search Engine by drewboy

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.