Because we have access to metadata that Google's naive crawler does not, we enjoy certain advantages when building a custom search. Certainly we can offer bells and whistles on the Super Search page that Google's advanced search can't match — they can't do filtering by author, ranking by node reputation, and so on.

I am confident that our users would find a KinoSearch-based Super Search considerably more usable than the current version, and that this would make them very happy. Programmers like to tweak tweak tweak. :) As a bonus, I also suspect that we can provide simple search results superior to what Google can offer, and certainly better than what we have now. It will be interesting to compare search results before and after we factor node rep into our ranking algorithm.

Whether or not it is worthwhile to maintain custom indexing and search for a public site depends on the site's size and the demands of its user-base. I expect that with several hundred thousand pages and extremely sophisticated users, we're well past the threshold. My guess is that the time it takes to maintain full-text search, including an advanced search interface, will be fully justified by a collective productivity increase. :)

SEO improvements to help web search engine spiders should probably be implemented regardless because increasing this site's visibility will aid people seeking answers to Perl questions from outside. However, I understand the powers-that-be have had good reasons for clamping down on spider access, historically.

A lot of these are obsoleted by a good ranking function, which will tend to pull the best hits to the top even without the additional metadata. For example, a search for "rectangular humphrey" turns up this: "I'm starting to get offers from people who want to sponsor features in my CPAN distro, KinoSearch," which is very relevant -- I didn't realize you were the author of KinoSearch, which you are also suggesting as a platform.

I agree that node ratings, etc., can be useful, but one of Google's big lessons is that quantity can beat quality: intelligent analysis of huge amounts of generic data can beat analysis of specialized data. This is particularly visible in its approach to natural language translation, but is nearly as important in search.

I didn't realize you were the author of KinoSearch, which you are also suggesting as a platform.

And I, in turn, was unaware that dmitri was my sock puppet. ;)

intelligent analysis of huge amounts of generic data can beat analysis of specialized data.

Sure, those techniques are powerful... The brute force "did you mean" stratagem^[1] is tough to top, no question!

As for whether we'll be able to deliver an overall improvement on the PerlMonks search experience, I guess we'll just have to present something, and people can vote with their clicks.

^[1] Major search engines decide what to suggest based on search history: what most people have typed in after misspelling something. This has proven superior to algorithms based on edit distance.

--
Marvin Humphrey
Rectangular Research ― http://www.rectangular.com

No joy.

So I went to advanced search, asked for the same items with a "from perlmonks.com only" qualifier and exact phrases using root node titles again.

Again, no joy!

My little experiment may have been inadequate in many ways, especially the limited sample.

But it may be that Google's indexing of pm is painfully inadequate. Running the same experiment against some of the sites I maintain turned up no such shortcomings, even when targeting a word used exactly once in a site that's been on line for barely a week.

And while crawling the links has costs -- for pm and for the user who has to wade thru '"Re^3: blah," "Re^2: blah",' the best material, IMO, tends to be in the replies, rather than the OPs. And wading thru the OPs and their answers, seeking the relevant ones can also be painful for the seeker.

I agree with the second part... some topics I want to search for may only be mentioned in the Re: nodes.

Maybe it would be convenient if the search results page had a "link to OP" next to the Re: results?

~dewey

The head node's page includes all the Re: nodes by default. I meant that Google should see a single page for each thread, rather than one starting at each subnode.