Web application performance monitoring tools

I've been building web applications for quite a few years now. One of the things I always feel lacking is the live web application performance monitoring. We spend lots of time (here and other sites) learning about various coding/design principles that help performance. But a live application behave quite differently from subroutine benchmarks/profiling. Often, when someone (e.g. a customer) alerts me of a inconsistent performance problem, I'm frankly at a loss as to (1) what is causing the problem, and (2) how to resolve it. Since such problems go away either after restart or under lower load, it's pretty difficult to look through code to catch it.

I'm wondering what you monks do for your live applications? What do you really do --as opposed to "I'll take a look"-- when someone tells you "hey, your app is really slow now"? Are there tools available? Can DProf being run in CGI/mod_perl and with how much overhead? Of couse you can log some timing for, say db calls, but it's too intrusive (requires littling the code with such logging.)

I've read some tools like those from Wily Technology for java application monitoring, I think they do some bytecode instrumentation to accomplish that. Does or can perl have such tools? There is a APM (Application Portfolio Monitoring) industry, does it serve the perl market?

Comment on Web application performance monitoring tools

Replies are listed 'Best First'.
Re: Web application performance monitoring tools by ghenry (Vicar) on Nov 28, 2005 at 09:14 UTC
Hi, There are a few I can think of that let you specify parameters etc. to test for, e.g. set and retrieve things on the page: Flood - a profile-driven HTTP load tester Performance test tools (list of) Stress Testing an Apache Application Server in a Real World Environment - Which mentions Flood As it's your app your testing, you know the parts of it that generate and except changes, new pages etc. so you can then at least get an idea if it's your code or your DB etc. Lastly, I think there is more to this than just your Perl code, it's the whole lot: Web Server Operating System Operating System Tuning Memory Disk I/O and Configuration Bandwidth High Availability Clusters Caching Plus lots more..... So depending on what factors you can check and change in reality, you might just start at the top, "Load Testing" and then work down. But like I said, this is a whole other field with somewhat of a "Black Art" associated with it. HTH. Walking the road to enlightenment... I found a penguin and a camel on the way..... Fancy a yourname@perl.me.uk? Just ask!!!	[reply]
Re: Web application performance monitoring tools by dragonchild (Archbishop) on Nov 28, 2005 at 15:42 UTC
My solution is very simple: I log everything. In particular, for every request, I log: Who made the request What was the request (the full URL) When did I receive it When did I send the response Were there any errors? I do this logging in before-request and after-response hooks. Catalyst, CGI::Application, Rails, and mod_perl in general all give you the ability to do these kinds of things. If you do that, you can very quickly tell when certain requests are taking a long time. You can also direct a user to their IT department if you're seeing that your side turns the request around in 2 seconds. More importantly, you can start do profiles based on time of day. On the first application I did this, we very quickly discovered the cause of one bottleneck - the number of requests between 1pm and 2pm was about 10x the number of requests between 12-1 and 2-3 combined. Turns out that there was an automated email from another application that was sent out at 1pm directing our users to log in to our app and do stuff. This meant that the legacy database we needed to get data from to coordinate with the app sending the email became our bottleneck. Because we could point that out with hard numbers (and graphs!), we got permission to mirror the legacy DB at 3am to our database instead of always hitting it. Our performance went up dramatically. (We needed permission because the mirror meant that information was now 24 hours old vs. being up-to-the-minute.) My criteria for good software: Does it work? Can someone else come in, make a change, and be reasonably certain no bugs were introduced?	[reply]
Re: Web application performance monitoring tools by brian_d_foy (Abbot) on Nov 28, 2005 at 20:12 UTC
Besides what other people have said, you can also monitor the resources your application uses directly (so you don't really need a perl monitor). Most significant slow-downs I've encountered involve the application waiting to use something (such as a database connection or physical memory (pagign)). Your basic sysadmin monitoring tools can help there. Once you have some real time (or close to real time) monitoring of the resources, you can match up activity in the web server log with the data from the other sources. -- brian d foy <brian@stonehenge.com> Subscribe to The Perl Review	[reply]
Re: Web application performance monitoring tools by perrin (Chancellor) on Nov 28, 2005 at 19:05 UTC
You can't reasonably run DProf on a live server. You can get basic timing information through a custom mod_perl log handler (as described in the docs) or some logging with Time::HiRes. A logging strategy is probably your best bet for this info. And of course a tool live Nagios or mon can do a good job of telling you when the site is slow before your users do.	[reply]
Re: Web application performance monitoring tools by radiantmatrix (Parson) on Nov 28, 2005 at 21:13 UTC
Of couse you can log some timing for, say db calls, but it's too intrusive (requires littling the code with such logging.) If your concern is "littering the code with such logging", the maybe you should check out Log::Log4Perl. Yes, you put many logging statements in the code, but their purpose is clear (you can say things like `$log->debug('starting DB call');`). Besides that, you can leave them off through a configuration directive, and only turn on the low-level logging when someone is reporting a problem. Without the ability to easily log verbosely, tracing down complex profiling problems is nigh-impossible. As an asside, I've had reasonable success with doing proactive performance monitoring. For example, I have put code into production that triggers a warning to a specific logfile if a DB query takes longer than a configurable number of seconds. A cron job than checks that logfile, and if several such warnings have occurred in a given time span (e.g. 20 warnings in 5 minutes), an e-mail is sent. Then I can work on fixing the issue before people start complaining. ;-) <-radiant.matrix-> A collection of thoughts and links from the minds of geeks The Code that can be seen is not the true Code "In any sufficiently large group of people, most are idiots" - Kaa's Law	[reply] [d/l]
Re: Web application performance monitoring tools by tilly (Archbishop) on Nov 29, 2005 at 06:29 UTC
If you're like me, your web applications do most of their work in a database. Unfortunately you simply can't turn on logging across the board for a database under load because if you do, then it falls over. (Well most of you can..but that is because your databases aren't very loaded by my standards...) With Oracle I've gotten good mileage out of turning tracing on for one Apache child's database handle. Combined with sending a few database calls whose only purpose is to add some basic logging (eg `select :1 as page_begin from dual`), the database trace contains a great deal of information about how the application is really spending its time. You'll have to investigate your database's options for yourself. Be prepared to wind up writing scripts to post-process the tracefile to mine useful information out of it.	[reply] [d/l]