There is a feature in Lotus Notes called a 'twisty' which is basically a control that allows a section of a document to be hidden or shown. The easiest way to think about this is a <readmore> tag. A twisty can contain further twisty sections etc. This works fine in a Lotes Notes client since the whole document is loaded and there are functions that allow you to expand all of the twisty sections.

In a web client (served by Domino R5) however the implementation of a twisty is less straight forward. The document that is outside of the twisty is displayed with each twisty being a link back to the original document. Each link has a set of arguments that describe which sections of the document should be shown. The argument structure looks something like:

This shows that sections 1, 2 and the first section inside of section 1 are expanded. Showing all of the sections is simply a question of constructing the correct argument. The problem is that the sections do not always start at 1 (there are often sections that are hidden from public view) and that there are an unknown number of sections, sub-sections, sub-sub-sections etc.

My proposed solution
I want to create a CGI script that acts as a lightweight proxy for Domino pages. The idea being that I can append the actual location of the content (full Domino URL) and do the twisty expansion by analysing the links in the document and requesting further documents until all of the twisties are expanded. The user (in this case the 'user' is actually a search engine spider) will just see the whole document without having to expand any twisties.

My first step has been to use LWP and HTML::LinkExtor to get the document from Domino and get the links. The further steps that I need to take are to process the links in order to compile the arguments for the new URL. This step will be repeated until the document is expanded. I will add additional functionality to handle authorization (basic and cookie based) when I get over the link parsing. The code so far is lifted from the HTML::LinkExtor POD and looks like this:

sub getLinks
{
    my $url = shift;
    my $ua = LWP::UserAgent->new;
    my $p = HTML::LinkExtor->new(\&callback);
    
    # Request document and parse it as it arrives
    my $res = $ua->request(HTTP::Request->new(GET => $url),
                  sub {$p->parse($_[0])});
    
    print  $res->code;
    # Expand all image URLs to absolute ones
    my $base = $res->base;
    
    @links = map { $_ = url($_, $base)->abs; } @links;
    return @links;
}

sub callback
{
    my($tag, %attr) = @_;
    return if $tag ne 'a';
    push(@links, values %attr);
}
[download]

Naturally - I have the usual feelings that a) this has been done before and b) there is a good chance that there is code out there already that solves the problem. Any help and advice is appreciated.

The real answer to the problem is to install and use Domino R6 which (as far as my research tells me) represents twisty sections as <DIV> sections with JavaScript doing the expanding / collapsing on the client browser. This solution is not available to me at this time since the number of Domino R5 servers is large and an upgrade is not planned for some time.

Thanks in advance.