Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??

Actually, PerlMonks runs on three FreeBSD boxes. We have one DB server and two web servers.

We had a lot of problems when the site was first moved from Linux to FreeBSD. It became clear (to me, anyway) that with our build of MySQL on FreeBSD (and this was back before the problem linked from MySQL on FreeBSD had been identified), a single long-running query could cause lots of other queries to block and that this was causing the site to completely lock up for several seconds or minutes (for example, whenever super search was used).

This problem was worked on at several fronts. vroom rewrote simple title search and super search to use full-text indexing, then I rewrote them both again. I remember reworking the backups repeatedly to get them to not lock up the site and reworking other maintenance tasks and writing breathe to get them to not lock up the site.

We all researched and eventually found the first reports of the problem and then possible fixes. ehdonhon rebuilt MySQL as suggested but the site just refused to stay up on it. We investigated switching back to Linux but never managed to get the resources needed for that.

I patched and recompiled Term::ReadKey so that it no longer was a CPU hog on FreeBSD so that we could use 'mytop' more often to monitor the DB. Then I reworked it and renamed it as sqltop and taught it how to run as a daemon, logging long-running queries and even killing ones that ran *too* long.

We finally had the site running pretty smoothly (though not exceedingly quickly).

Then pair.com gave the DB server a huge upgrade (I think the CPU speed quadrupled) and the site was pretty happy for a while... though the traffic volume continued to grow...

Last January the site was the busiest it had ever been. Traffic actually fell off a bit after that. This January we hit another peak and finally the web servers were having a hard time keeping up.

Luckily I'd been working on finding problems and fixing them so I had several patches to apply that would eventually help fix the problem and really help quantify how well we were fixing things.

Unfortunately, trying to deploy these was increasingly difficult because the web servers were often overloaded. I even ended up backing out a big set of behind-the-scenes changes even though my testing showed they increased web server load by less than 5%, because the site became nearly unusable for part of the next day.

And we started getting the occassional web server needing to be rebooted. So I was trying to put together some data to back me up so I could ask Pair for a third web server without feeling like I was just whining... when Pair said the web servers were due hardware upgrades. Yeah!

The web servers were certainly getting bogged down some of the time (you could easily notice this when logged into them). The load averages were higher than I expected (but usually not *really* high), but in any given 5-second period the web servers usually had some spare CPU cycles (usually quite a few). And they were only needing a fraction of their memory. My rather wild guess was that the web servers were alternating between everything waiting while MySQL got locked up for just a second or less and then lots of queries completing at once and being CPU starved to process them all.

In any case, today's doubling of the web server's CPUs from 1GHz to 2GHz is very welcome.

Fairly soon we should be testing MySQL v4 w/ LinuxThreads which should allow us to make better use of the DB server's power. And I'll be deploying efficiency improvements for quite a while. And demerphq just did a ton of work to make the best/worst nodes lists much more efficient (and other improvements). So the site might be quite happy for another year of growth in popularity.

Anyway, I wanted fill in a bit about the site's performance history. I hope that was informative.

- tye        


In reply to Re^3: Web servers get hardware upgrade (PM hw) by tye
in thread Web servers get hardware upgrade by tye

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others admiring the Monastery: (3)
As of 2024-03-29 14:50 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found