I deliberately kept the detail down to reduce the size of the initial question. However I'm more than happy to give more details if people are interested.

The situation is as follows:

We have a backend application which listens on a particular port (not that it matters, but this is configurable). The message format is simple. Messages are supposed to each have a unique message number (but this is determined by the client). The message itself is a pipe-delimited format, which consists of a message header and one or more segments of data. There are different types of traffic in this protocol, for example, information (handshakes), keep-alives and data packets (of pipe delimited messages).

All I'm interested in is providing a HTTP interface to this protocol. The HTTP client is a fixed implementation (cannot be rewritten easily, it is legacy code), as is the backend system (which is closed source). Suggestions such as SOAP would be excellent, but don't fit well unfortunately. We basically need a "request-response" HTTP server which talks (somehow) to the backend system. The backend system works in exactly the same way ("request-response"), it just uses a different protocol. There is really no concept of "sessions" or logging in, so it is well suited to presentation over HTTP.

We have limited time (as always), so I've managed to persuade my superiors that Perl should be the development language of choice (I know a number of languages, but Perl is my strongest). Despite being a fairly simple program, it will be revenue generating, so needs to be pretty stable and robust. The volume of transactions will be quite high, so something which forks off a new server for each request would pretty soon die I think.

What I was really asking for were general approaches, perhaps pointers to more information (which everyone has already kindly given). The one approach I liked is using mod_perl, and maintaining a persistent connection to the backend system (a-la Apache::DBI). Besides reading the Apache::DBI source (which I might do), is there a general approach for doing this mod_perl "once-at-startup" connection pooling?

There are numerous approaches (e.g. IPC with shared directory structures, databases, CGI scripts), but I was interested in other people's slant on this problem.

Thanks again for everyone's time.

$q=[split+qr,,,q,~swmi,.$,],+s.$.Em~w^,,.,s,.,$&&$$q[pos],eg,print

One other suggestion in addition to the others mentioned so far - POE.

I've used POE to provide an HTTP API to a legacy protocol with persistent, stateful connections. It worked well for me. I actually expected to have to rewrite it in C once it had been prototyped, but it coped with the load well. Might be worth investigating.

We had a POE process that sat their taking HTTP requests from a mod_perl server that was handing the application logic, and mapped that onto a pool of connections to the backend server.

As for the HTTP API - you might want to consider trying to twist your stateful protocol into something in a more RESTful style. Then you'll be able to deal with scaling issues with more traditional web based caching/proxies.

For example, in my project some of the information passed over protocol was only stateful (and slow) because of it's history a something that ran over a serial line. Once we got the data out we could map it onto a stateless set of URL referenced documents that could then be proxied and cached - reducing the load on the POE backend server considerably and allowing it to spend most of its time on the data that had to have state.