we are aiming for a high availability cluster.
Even though the network stack creates a bottleneck between servers, at 1gbit it should be small, but the benefit is redundancy - a server can fall over and we can continue service on another server. But the rub is - how to quantitively state what performance loss or gain there is in the n-tier setup.
I can see that you can increase the number of concurrent users, but at what cost in the slow down across the network between db server and web server.