jimbus,
Sorry I can't help with most if this ... but

The other issue I have is the knowledge base. How do I build a document db with lucene, but tell the search engine that only certain roles can see certain docs?

The way we handle that is to pass an extra search param that reduces the result set to that role. For instance, to return all docs containing the word bar that only the foo department can see, your query may look like this: dept:foo AND all:bar.

You just need to ensure there's no back doors into your search and that all queries pass the appropriate role param.

Update:Forgot to mention it but just as important you would need to create your index with those role appropriate fields.