Of couse you can log some timing for, say db calls, but it's too intrusive (requires littling the code with such logging.)

If your concern is "littering the code with such logging", the maybe you should check out Log::Log4Perl. Yes, you put many logging statements in the code, but their purpose is clear (you can say things like $log->debug('starting DB call');). Besides that, you can leave them off through a configuration directive, and only turn on the low-level logging when someone is reporting a problem.

Without the ability to easily log verbosely, tracing down complex profiling problems is nigh-impossible.

As an asside, I've had reasonable success with doing proactive performance monitoring. For example, I have put code into production that triggers a warning to a specific logfile if a DB query takes longer than a configurable number of seconds. A cron job than checks that logfile, and if several such warnings have occurred in a given time span (e.g. 20 warnings in 5 minutes), an e-mail is sent. Then I can work on fixing the issue before people start complaining. ;-)