My solution is very simple: I log everything. In particular, for every request, I log:

• Who made the request
• What was the request (the full URL)
• When did I receive it
• When did I send the response
• Were there any errors?

I do this logging in before-request and after-response hooks. Catalyst, CGI::Application, Rails, and mod_perl in general all give you the ability to do these kinds of things.

If you do that, you can very quickly tell when certain requests are taking a long time. You can also direct a user to their IT department if you're seeing that your side turns the request around in 2 seconds.

More importantly, you can start do profiles based on time of day. On the first application I did this, we very quickly discovered the cause of one bottleneck - the number of requests between 1pm and 2pm was about 10x the number of requests between 12-1 and 2-3 combined. Turns out that there was an automated email from another application that was sent out at 1pm directing our users to log in to our app and do stuff. This meant that the legacy database we needed to get data from to coordinate with the app sending the email became our bottleneck. Because we could point that out with hard numbers (and graphs!), we got permission to mirror the legacy DB at 3am to our database instead of always hitting it. Our performance went up dramatically. (We needed permission because the mirror meant that information was now 24 hours old vs. being up-to-the-minute.)