Swish-E supports retrieval of docs through HTTP rather than on
the filesystem. Take a look at the
Spidering section of the manual.
In fact, the implementation is done through LWP. I haven't
looked at it much other than to notice that, though, although
I do remember thinking at the time that it seemed a bit kludgy
to have to invoke a Perl script to spider the site. Presumably
there's quite a bit of interaction between the main C source,
the system, and the Perl program that could be solved by
having an actual HTTP implementation inline.
