I think you'll find Harvest to provide the web crawling, indexing and the web search UI you're looking for.

It's very scalable (can be clustered for both search and crawl IIRC) is pretty mature, and is actively developed.