Background

I'm setting up a pair of web servers to have failover redundancy. If the main server fails, the backup server takes over.

I'm trying to automate testing of my prototype. I figure I'll need two pieces, a "observer" that sits on a client machine and watches and coordinates the action, gathers results etc... Then I'll need a "gremlin". There will actually be two gremlins, one on each server. The gremlins will cause failures and forward internal information to the observer.

Actual Question

Whats a good way to facilitate the communication between those three guys? Quick and easy is my highest priority. This will only run in my test lab (I don't want gremlins causing failures on my real servers) so stability and security aren't a big priority.

The observer needs to be able to tell the gremlins when to cause failures. The gremlins need to confer among themselves to coordinate the failures. The gremlins also need to be able to tell the observer all kinds of status info about the server that gremlin is running on.

Thanks Monks!

-Pileofrogs

Updates: