Because we have access to metadata that Google's naive crawler does not, we enjoy certain advantages when building a custom search. Certainly we can offer bells and whistles on the Super Search page that Google's advanced search can't match — they can't do filtering by author, ranking by node reputation, and so on.

I am confident that our users would find a KinoSearch-based Super Search considerably more usable than the current version, and that this would make them very happy. Programmers like to tweak tweak tweak. :) As a bonus, I also suspect that we can provide simple search results superior to what Google can offer, and certainly better than what we have now. It will be interesting to compare search results before and after we factor node rep into our ranking algorithm.

Whether or not it is worthwhile to maintain custom indexing and search for a public site depends on the site's size and the demands of its user-base. I expect that with several hundred thousand pages and extremely sophisticated users, we're well past the threshold. My guess is that the time it takes to maintain full-text search, including an advanced search interface, will be fully justified by a collective productivity increase. :)

SEO improvements to help web search engine spiders should probably be implemented regardless because increasing this site's visibility will aid people seeking answers to Perl questions from outside. However, I understand the powers-that-be have had good reasons for clamping down on spider access, historically.

--
Marvin Humphrey
Rectangular Research ― http://www.rectangular.com

In reply to Re^2: Running SuperSearch off a fast full-text index. by creamygoodness
in thread Running SuperSearch off a fast full-text index. by dmitri

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.