Try load testing your site. It sounds like what you are saying is that in development / test the server behaves just fine. If this is the case, use a load testing suite to test the number of clients that work well. Use your server logs to determine the max number of clients you are seeing in production, and ramp from your testing values (2-3) up to that number.

If your server is set to only accept a certain number of connections, it will not necessarily show a higher load when clients are "waiting in the wings".

Also, if you have multiple instances, each one has its own connection pool, so you may not see the slowdown on both.

For example, assume that your server is set to accept 10 simultaneous connections. If you have 2-3 clients, you will not see any slowdown. If, however, you have 15 clients make request, each one of them uses keep-alive to request associated graphics (10), js (2), and css (1) files, and each request from a client takes a round trip time of .5 seconds, each client will hold a connection for 7 seconds. So the first 10 clients hold connections for 7 seconds, then the next five get to try. The second set of clients see the 7 second delay, plus their own 7 seconds for their requests, for a total of 14 seconds.

Now imagine that you have 2 requests coming in per second. That is 10 requests (the server max) every 5 seconds, but you are only able to handle 10 every 7 seconds. Your wait queue will start to stack up, and your clients will see longer and longer delays.

When your server sees fewer request per second, it gets a chance to catch up, which is why you sometimes see good performance, and other times you do not see good performance.