If you're like me, your web applications do most of their work in a database. Unfortunately you simply can't turn on logging across the board for a database under load because if you do, then it falls over. (Well most of you can..but that is because your databases aren't very loaded by my standards...)

With Oracle I've gotten good mileage out of turning tracing on for one Apache child's database handle. Combined with sending a few database calls whose only purpose is to add some basic logging (eg select :1 as page_begin from dual), the database trace contains a great deal of information about how the application is really spending its time.

You'll have to investigate your database's options for yourself. Be prepared to wind up writing scripts to post-process the tracefile to mine useful information out of it.