Have you profiled your application using Devel::NYTProf::Apache (Profiling mod_perl via Devel::NYTProf) or similar profiling/benchmarking tool?

Is the nature of the requests such that they can be satisfied by the CPU alone, without doing any time-consuming I/O?   Or, is the I/O capacity well-balanced against the workload such that it can service whatever demand it receives without building up lengthy queues?   If so, “100% CPU utilization” might well be cause for celebration.

The operating system will always push for 100% utilization of all resources, if the demand is present.   Usually, some bottleneck or another pushes-back against that goal such that you can’t actually get there, and the processes start to accrue unwanted involuntary wait-time with a resultant loss of overall system efficiency against its theoretical capacity.   So, what you really want to be looking for are those involuntary wait-times ... and you want to look for them in terms of the requests, not [just...] the hardware.

If I may draw an analogy to an ordinary business, there are always two different trouble-metrics that management has to consider:

1. The efficiency of its own warehouses, trucks, manufacturing centers and so on, so that people who could be working are not forced to be standing around.
2. The life-cycle of a particular customer order from the moment that the order is placed to the moment that the last now-empty truck leaves their loading dock ... squeezing out any unnecessary and avoidable delays that any such order actually incurred.

The second of these two is what will cause the customer to take his business elsewhere.   Employees can indeed be 100% busy doing useful things and still be delivering service that the customer will judge to be inferior.