Hmm...since I use a separate server for images and etc, I guess I'll try disabling keep-alive. My server is set to accept a pretty large number of connections though, and if anything I would assume the CPU process would be the on reaching 100% and not be around 30%. The network card also seems to be using only 2%.
What is weird though is that I have things take 200-1000 milliseconds and then out of nowhere a request takes 16000. And it is not like that page is anything special cause for others it takes 900 for example. Also, there is no pattern of same IP having long load :/..so it is kinda irregular, which is making it pretty difficult to debug. Then at one point it slows down more and more till everyone takes 10min per page(and no I am not exaggerating here, I had at one point it show over 1 million milliseconds per request)
Comment on Re^5: Timing concerns on PerlEX/Mod_Perl
If your symptom consistently manifests at 30% CPU, then perhaps you're hitting a CPU utilization limit rather than a simultaneous connection limit. There are also limits on most OSes for number of files open per process and amount of memory per process or process tree. There are some limits on some OSes on a per-user rather than per-process basis, too. Make sure you're not hitting any of those.