I'd certainly try to prototype this first with something like FCGI plus some high performance webserver - probably nginx.

There are so many potential problems already solved, so much know-how already available in the human cloud.

Like concept to scale up with a server farm...

"Standing on shoulders of giants" and so on...

And if this "prototype" doesn't deliver sufficiently, one can still learn from the concepts when implementing the DIY server.

Saying so, I'm still in doubt the OP has really identified the bottlenecks of his existing solution.