in reply to Re: CgiProxy And Heavy Visitors
in thread CgiProxy And Heavy Visitors

thanks !
with this server this folowing was my max records in apache :
mod_perl 30 request per second + automaticly restart of apache each 10 minutes ! becouze of low memory !
mod_cgi 45 request per second !

its very hard and need high money for load balancing ! i requested my Datacenter to add 3gb extra memory
so 1gb+3gb=4gb is 4gb enough ?! or i am wrong ?
in mod_perl i just had memory problem and server load was under 0.50
i am running just cgiproxy on this server and nothing

Replies are listed 'Best First'.
Re^3: CgiProxy And Heavy Visitors
by jhourcle (Prior) on Apr 10, 2005 at 21:17 UTC

    You haven't been very specific on exactly how many different scripts you're running, or what those scripts are doing that might require them to use so much memory. That information could result in a significantly different recommendation.

    Without knowing just what's going on, I have no idea if 4GB is enough ... or what even qualifies as 'enough'. Making sure it's not going to be paging under its current load is one thing, but I'm guessing your company won't be happy if you creep back up over the line before the year's out. (and I'm guessing you will, with any significant growth, given that you're restarting mod_perl every 10 minutes.)

    I'd suggest that you do more than just throw memory at the problem, and try to determine what the real problem is. I remember once being rushed to the emergency room, and after many hours there, being released and told to take a pain killer, because they had no idea what was wrong. (it took another 18 months and 3 different doctors, after a whole bunch of other medical problems before I figured out on my own that I had become lactose intolerent). If I had listened to the 'professionals', who had a vested interest in my coming back more times and giving blood samples and the like, (as they'd stop getting money if I didn't keep showing up), I'd probably be miserable to this day, as it felt like my innards were ripping itself apart, and couldn't travel without... let's just say 'many unpleasent problems', that we'll compare to your restarting every 10 minutes.

    No one has been given enough information to make an authoritative recommendation to solve your problem with anything more than a band-aid to stabalize it so that you can do some more in-depth analysis to find the real problem. Writing software can be an interative process, and as you find problems, they can be fixed, or as the parameters change (in your case, the number of hits per minute), the program may have to be changed to deal with conditions that it wasn't expecting when initially written.

    The problem with mod_perl is that the program is left running through many, many callings. If you put a program with a memory leak under mod_perl, it will leak with every calling, until it occupies all of the physical memory on the system, and starts paging. It's also possible that you just a lot of different programs on your system, and they call many varied modules, and some may use a fair amount of memory, so that as the apache child processes run each of the scripts, they use more memory (not necessarily a leak, just not a candidate to be using mod_perl)

    Without having seen your configuration, or knowing anything other than what you have said, I will state that I do not believe that placing more memory in the server will be a long term fix. Just like putting more oil into a car that's leaking only helps for so long, you're very likely to run into this same problem in the future.

    I bitched at BrainBench for their 'Webserver Administrator' test, because they had questions where they gave limited details and asked 'what is sure to fix the problem', and gave solutions like 'adding more memory', 'use RAID', and the like ... this is a similar situation. There is no silver bullet fix to every situation, and there are too many unknowns to assure you that any given solution will fix the problem.

Re^3: CgiProxy And Heavy Visitors
by tilly (Archbishop) on Apr 10, 2005 at 22:42 UTC
    The load refers to how many processes are ready to run on average when the scheduler has to pick someone. If the system is on its knees and load is low, then you have a bottleneck elsewhere. It seems likely that your system is heavily in swap, but it is also possible that you are doing something else silly like disconnecting from the database with every request. (That particular silliness should not bring you below the performance of mod_cgi, and you show other signs of memory problems.)

    I would still go with my previous recommend as an obvious first step. After you do that you should be able to drop the number of mod_perl Apache kids substantially and maintain throughput. It would also be good longer term to audit the code looking for where memory is likely to be allocated and not freed at the end of each request. Devel::Leak may help you do that.

    I'm not entirely sure of what you mean by "automatically restart of apache every 10 minutes". If you mean that Apache has to stop and launch again, that suggests that your machine is getting out of memory and random processes are being killed. This is very bad. If you mean that Apache children are dying that frequently, that is fairly harmless. Think of it this way, if your average Apache child lasts 10 minutes and you have 20 of them, then you have one child being launched every 30 seconds. Compare with 45 being launched per second and you see improvement. The time to be able to launch processes is not going to be an issue until they die a lot faster than that!

    Incidentally once you are on mod_perl, writing real mod_perl handlers rather than using Apache::Registry should both considerably boost performance and save memory. (I don't know that you're using that, but it is a pretty safe guess.)

      hi again i installed 4gig ram and now i am useing mod_perl now i dont have any problem ! cpu load low , memory usage low and pages loading very fast i just wana know how can i configure perl to use mod_gzip , already installed mod_gzip and it compressing all pages but its not compressing perl scripts :( what can i do ?
        I've never tried to make mod_perl and mod_gzip cooperate. My natural inclination would be to go with the reverse proxy setup that I suggested, and put mod_gzip on the proxy servers. If that fails my other alternatives that I'd consider would be to google to see if anyone else has done it, post the question to Seekers of Perl Wisdom (where more people will see it than will see this reply), ask on an Apache list, or try to upgrade to Apache 2 and hope that mod_deflate works better than mod_gzip. I'd try them in that order.
      hi again now i have 4gig memory useing mod_perl and server is fast ! just some more question : 1-can u recommend a good number for MinSpareServers,MaxSpareServers,StartServers and MaxRequestsPerChild 2-how can i turn on mod_gzip for perl scripts ? thanks