Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Web servers get hardware upgrade

by tye (Sage)
on Feb 13, 2004 at 20:10 UTC ( [id://328867]=monkdiscuss: print w/replies, xml ) Need Help??

Pair informed me that our web servers were due for hardware upgrades and asked when would be a good time to do that. It looked to me like Friday afternoons were the least busy time to do such based on the window available... and timing ended up with this happening Friday morning.

So the upgrades will happen at 4pm Eastern Time today (about 1 hour from now). I asked for more details (how long will it take, what is being upgraded) but haven't heard back yet and I don't want to delay this announcement any longer. I'll add details when I get them.

- tye        

Replies are listed 'Best First'.
Re: Web servers get hardware upgrade
by ehdonhon (Curate) on Feb 13, 2004 at 21:36 UTC

    Upgrades are finished. Both webservers got upgraded to the present QS3 Specs.

      Interesting.

      I've never really considered what hardwware PM runs on, but in one day I find out it's running on a FreeBSD box -- from the same person that posted a link in the chatterbox to info about the performance of MySQL on FreeBSD

      Makes me wonder if the MySQL setup provided by pair is running LinuxThreads.

      Clarification: I have no opinions or objections to the hardware/OS PM runs on. I was just geniunly curious how our MySQL server was compiled. Based on tye's reply, it looks like they'll be testing out hte LinuxThreads option soon.

        Actually, PerlMonks runs on three FreeBSD boxes. We have one DB server and two web servers.

        We had a lot of problems when the site was first moved from Linux to FreeBSD. It became clear (to me, anyway) that with our build of MySQL on FreeBSD (and this was back before the problem linked from MySQL on FreeBSD had been identified), a single long-running query could cause lots of other queries to block and that this was causing the site to completely lock up for several seconds or minutes (for example, whenever super search was used).

        This problem was worked on at several fronts. vroom rewrote simple title search and super search to use full-text indexing, then I rewrote them both again. I remember reworking the backups repeatedly to get them to not lock up the site and reworking other maintenance tasks and writing breathe to get them to not lock up the site.

        We all researched and eventually found the first reports of the problem and then possible fixes. ehdonhon rebuilt MySQL as suggested but the site just refused to stay up on it. We investigated switching back to Linux but never managed to get the resources needed for that.

        I patched and recompiled Term::ReadKey so that it no longer was a CPU hog on FreeBSD so that we could use 'mytop' more often to monitor the DB. Then I reworked it and renamed it as sqltop and taught it how to run as a daemon, logging long-running queries and even killing ones that ran *too* long.

        We finally had the site running pretty smoothly (though not exceedingly quickly).

        Then pair.com gave the DB server a huge upgrade (I think the CPU speed quadrupled) and the site was pretty happy for a while... though the traffic volume continued to grow...

        Last January the site was the busiest it had ever been. Traffic actually fell off a bit after that. This January we hit another peak and finally the web servers were having a hard time keeping up.

        Luckily I'd been working on finding problems and fixing them so I had several patches to apply that would eventually help fix the problem and really help quantify how well we were fixing things.

        Unfortunately, trying to deploy these was increasingly difficult because the web servers were often overloaded. I even ended up backing out a big set of behind-the-scenes changes even though my testing showed they increased web server load by less than 5%, because the site became nearly unusable for part of the next day.

        And we started getting the occassional web server needing to be rebooted. So I was trying to put together some data to back me up so I could ask Pair for a third web server without feeling like I was just whining... when Pair said the web servers were due hardware upgrades. Yeah!

        The web servers were certainly getting bogged down some of the time (you could easily notice this when logged into them). The load averages were higher than I expected (but usually not *really* high), but in any given 5-second period the web servers usually had some spare CPU cycles (usually quite a few). And they were only needing a fraction of their memory. My rather wild guess was that the web servers were alternating between everything waiting while MySQL got locked up for just a second or less and then lots of queries completing at once and being CPU starved to process them all.

        In any case, today's doubling of the web server's CPUs from 1GHz to 2GHz is very welcome.

        Fairly soon we should be testing MySQL v4 w/ LinuxThreads which should allow us to make better use of the DB server's power. And I'll be deploying efficiency improvements for quite a while. And demerphq just did a ton of work to make the best/worst nodes lists much more efficient (and other improvements). So the site might be quite happy for another year of growth in popularity.

        Anyway, I wanted fill in a bit about the site's performance history. I hope that was informative.

        - tye        

Re: Web servers get hardware upgrade
by Anonymous Monk on Feb 25, 2004 at 23:01 UTC
    I choose pair for our webhosting almost solely on their hosting perlmonks. (And noted it when I signed up)

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: monkdiscuss [id://328867]
Approved by The Mad Hatter
Front-paged by b10m
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others sharing their wisdom with the Monastery: (4)
As of 2024-03-29 12:23 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found