streaming SOAP::Lite service ?

erroneousBollock has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

I've developed a perl webservice framework (internal consumption only for now) which maps webservice handlers to perl classes.

Currently it supports:

SOAP services
JSONRPC services
static and dynamic content, read at runtime.

Sometimes, method calls to SOAP or JSONRPC services return large amounts of data (eg: 10,000 "records" or more). In these cases a significant amount of time elapses between return of values and the sending of data to the client - mostly down to XML encoding, sometimes due to latency between the webservice and some external datasource.

I'd like to be able to "stream" my return values to the client (and for now I'm focussing on SOAP). That's no problem for the client-side, as I can simply implement an incremental reader with a SAX-based XML parser - I know what each "record" will look like based on the WSDL.

A natural solution to the server-side part would be that:

the method returns a service "continuation"; SOAP::Lite could create a stub XML document based on the (empty) returned list, but should only immediately send up to the <body> tag to the client.
the server could periodically "call" the continuation to obtain batches of "records"; each returned record should be turned into the appropriate XML fragment then sent to the client immediately.
when the continuation returns undef, the server should send the closing tags from the stub XML document to the client.

My questions are threefold:

Is this the wrong approach?
If not, has someone done this before?
If so, am I missing some hooks in SOAP::Lite which makes this easier?

-David

PS: I'm aware of continuation-based web transactions in Seaside (smalltalk).

Comment on streaming SOAP::Lite service ?

Replies are listed 'Best First'.
Re: streaming SOAP::Lite service ? by jhourcle (Prior) on Mar 26, 2007 at 12:41 UTC
To the best of my knowledge, streaming is not possible with SOAP::Lite. Because you return your completed data structure to SOAP::Lite, it doesn't begin processing the result until that point. For some types of deserialization (eg, RPC-encoded arrays), it needs the entire data structure to before it can begin emitting the XML string If you were to use an NPH CGI script (because if it's parsed by the webserver, it's going to wait 'till it knows the file size so it can insert it in the HTTP header), and document/literal encoding, you could probably emit the XML as the string is being serialized. It's been a while since I've done serious digging through SOAP::Lite's serializer, so I don't know how easy it would be to piggyback on their serializer (so that you don't have to deal with the headache of supporting multiple versions of XML Schema on your own)... this might be something that you could do, and provide as an alternate SOAP toolkit, or try to coordinate into Byrne's plans for SOAP::Easy.	[reply]
Re^2: streaming SOAP::Lite service ? by erroneousBollock (Curate) on Mar 28, 2007 at 02:23 UTC
The more digging I do, the more I agree - SOAP::Lite doesn't have streaming in mind :-( At this point it seems that (without too much trouble) I should be able to get a SOAP::Lite subclass to serialise a 'stub' document, and a SOAP::Transport subclass to send only part of the stub to the client; I can do the same for the closing tags from the stub. I could probably treat each 'yield' of the continuation like a 'return' from the dispatched method -- proceed to serialise as normal -- but then somehow pull out the 'payload' of the serialised document as a fragment. I guess the hard parts will be discovering how to: create a re-entrant generic SOAP::Transport subclass, which will work such that it can be injected into the inheritance chain of the existing SOAP::Transport subclasses. do 'continuations' in a natural way in dispatched methods. If possible, I'd like to keep the usage of 'streaming' as transparent possible, so that the user of my toolkit can easily select whether to stream results -- either at compile time, or at runtime (perhaps based on some preliminary calculation of result-set size). While I am concentrating on SOAP for the moment, I'll also want to look at doing all this for JSONRPC... but seeing that it's modelled after XML::RPC which is similar in structure to SOAP::Lite, I guess that shouldn't be too much of a problem. -David.	[reply]