Persistent HTTP connection

mst226 has asked for the wisdom of the Perl Monks concerning the following question:

Hi everyone.

I am trying to write a web client that connects to an API using HTTP. The API server is the Gephi software (https://gephi.org) with the "streaming plugin" which accepts commands to modify a graph in real-time. My goal is to visualize Internet traffic showing end-stations as nodes and connections as edges.

The API was not written by me and there is no way to change its behaviour. (There might be, but this is out of scope)

This API is somehow special since the HTTP connection stays open the whole time (even if it is idle).

So, my plan is to:
1. make a http connection to the server
2. send commands (POST) in JSON format using the established connection
3. wait for a reply which might take a while
4. send more commands
5. wait for replies
and so on.

Only steps 1. and 2. are fixed. There is no order in the following steps. Some commands expect a reply while others do not. Also it is possible to send hundreds of commands without waiting for reply.

I tried to build the web/api client using LWP and IO::Socket::INET, but the connection is closed at some point in time.

Basically it is possible to open/close the connection for every single command, but this limits the performance of the web/api client.

The API is described here:
https://wiki.gephi.org/index.php/Graph_Streaming
Is it focused on Java but on the bottom there are examples of the command syntax.

Can you give me some hints how to continue?

Thanks.
-Markus

Comment on Persistent HTTP connection

Replies are listed 'Best First'.
Re: Persistent HTTP connection by jellisii2 (Hermit) on Apr 17, 2014 at 11:27 UTC
If the served UI behaves from a browser WWW::Mechanize::Firefox may be a good alternative to what you're doing.	[reply]
Re: Persistent HTTP connection by RonW (Parson) on Apr 17, 2014 at 16:01 UTC
Web servers often close connections for various reasons, so your client application has to be prepared to open a new connection as needed, probably also retrying commands. Retrying commands can be tricky if they don't always get a response. Perhaps part of the problem is sending too many non-response commands consecutively.	[reply]
Re: Persistent HTTP connection by pvaldes (Chaplain) on Apr 21, 2014 at 21:54 UTC
If you want to visualize nodes and edges you'll probably want to add graphviz to your plan	[reply]
Re^2: Persistent HTTP connection by mst226 (Initiate) on Apr 25, 2014 at 12:30 UTC
Hello pvaldes. Thanks for your answer. Graphviz is an excellent tool which I already use for other projects. In this case my focus is on real-time visualization. The information, which become nodes and edges is only known at runtime. I think that GraphViz is not capable of that, but not 100% sure...	[reply]
Re: Persistent HTTP connection by mst226 (Initiate) on Apr 25, 2014 at 12:36 UTC
Unfortunately I can't find a good solution for this, so I make every API call a separate http connection using LWP. That was exactly what I did not wanted, but I turns out that my client/server hardware performs at 160 API calls per second. Hopefully this is enough...	[reply]