The search engine that I use to index the content has a connector for Lotus Notes that sucks the content out using the Notes API. This works very well for Domino servers that are hosted on our internal network.

The Twisty problem still causes an issue when a document from a user search is viewed. At this time, the doc is generated as HTML by Domino and streamed through the search engine so that keyword highlights can be added etc. The users get annoyed when the search engine returns results that they can't immediately see because they are hidden in a twisty.

Another problem is that there are a large number of Domino servers hosted elsewhere (external hosting provider etc.) that I cannot connect to using the Notes API. These servers must be indexed via HTTP and the representation of the 'ExpandSection' argument plays havoc with the indexing. The search spider views each version of the URL as a new document.

The 'ExpandAll' argument is something that Domino web developers have been asking about for years. Lotus just couldn't do it in R5 and it can't be done using LotusScript. The Perl app that I am developing is the most recent in a long line of potential solutions.