My solution is very simple: I log everything. In particular, for every request, I log: I do this logging in before-request and after-response hooks. Catalyst, CGI::Application, Rails, and mod_perl in general all give you the ability to do these kinds of things.

If you do that, you can very quickly tell when certain requests are taking a long time. You can also direct a user to their IT department if you're seeing that your side turns the request around in 2 seconds.

More importantly, you can start do profiles based on time of day. On the first application I did this, we very quickly discovered the cause of one bottleneck - the number of requests between 1pm and 2pm was about 10x the number of requests between 12-1 and 2-3 combined. Turns out that there was an automated email from another application that was sent out at 1pm directing our users to log in to our app and do stuff. This meant that the legacy database we needed to get data from to coordinate with the app sending the email became our bottleneck. Because we could point that out with hard numbers (and graphs!), we got permission to mirror the legacy DB at 3am to our database instead of always hitting it. Our performance went up dramatically. (We needed permission because the mirror meant that information was now 24 hours old vs. being up-to-the-minute.)

My criteria for good software:
  1. Does it work?
  2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?

In reply to Re: Web application performance monitoring tools by dragonchild
in thread Web application performance monitoring tools by johnnywang

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.