After much sitting here and scratching my head, I guess I will have to turn to my fellow monks for some help.

First of all, let me explain my project. I am writing a search engine for a database of Campus Newspaper articles dating back to 1925 and only up to 1980. As of right now, it is about 60,000 rows long. It will most likely grow even bigger in the future.

About a week ago I completed the project and it worked pretty good, until I tried to search for a word that had more than a few thousand hits. Let me quick briefly explain how things are done right now.

My sql statement is:
my $searchstring = 'SELECT * from Exponent WHERE ( Description LIKE '%$tempword%' OR Article LIKE '%$tempword%' ) AND ( Date > '$datefrom' ) ";

This pulls everything down that contains even the smallest match, then I have perl loop over the results and weight them by the number of times the word is contained in the article title and article description.

This works fine until I try to pull down 30k rows. My question is this, is it possible in SQL to somehow write a function that takes 3 parameters ( a pattern to match, and a startslice and endslice so I only have to pull down the 10 or 20 matches I want to look at, not the whole thing?

Im afraid I am still learning perl, and my SQL is at even a lesser level than my perl understanding. Sorry for this rather long winded question, I hope I make sense :P

In reply to Inefficient search algorithm? by ruhk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.