There's a document on the apache website regarding performance, which could be a good place to start
davis

This is quite a wide subject but generally performance tuning can be most easily performed at an app level. However, first off I would recommend using apps like vmstat, mpstat, top, etc (assuming they are running something Unix like) and check for the four common system bottlenecks, CPU utilization, physical RAM/Swap utilization, network bandwidth and disk spindle utilization time. Generally one or more of these will indicate the source of the problem (which are usually bad SQL code, badly laid out database tables on physical spindles or very inefficient Perl code in that order of likeliness). Good luck :-)

Maybe looking at OnLAMP would be a good start!

It's probably not a big help at the moment, but if your friends had mock pieces of the application for testing, they could isolate each part in turn and do benchmarking that way.

For example, when testing something like a libnet module, it's a whole lot easier to whip up a fake Socket module that does exactly what you want than it is to work with all sorts of network configurations. So they could have a fake database object, a fake web server object, and so forth. The performance isn't quite real-world, but if they have enough fake pieces, they can benchmark each real piece in isolation. That can be handy.

Hey, testing can't solve everything, but this is a place it can help.

Thanks to all of you. Here's a quick summary of the answers I got:

performance problems in a LAMP system

The problem could be located in one or more components:

1. hardware/os
2. apache
3. mysql
4. the application itself

the usual suspects are the application itself (bad SQL code, inefficient Perl code) and the database (badly laid out database tables on physical spindles)

hardware/os

check: CPU utilization, physical RAM/Swap utilization, network bandwidth and disk spindle utilization time.
tools: vmstat, mpstat, top,...

apache

links: Apache Performance Tuning

mysql

tools: the mysql slow log, DBIx::Profile
links: linuxmag article on mysql performance tuning

the application itself

tools: Devel::DProf, Apache::DProf

2001-12-13 Edit by Corion : Added missing <OL> tag

Here's the right direction: Devel::DProf (or Apache::DProf for mod_perl). That will tell you which part of your application is slow.

The most common cause of performance problems in simple web systems is database performance, often due to poor table design or inefficient SQL. You can use DBIx::Profile to find slow SQL statements that need to be fixed.

Of course, before you do anything else you should make sure that your machines have enough RAM and are not swapping. The mod_perl guide you mentioned can help you deal with that.