Film at 11!

PerlMonks has been undergoing some issues lately, and we're doing our best to deal with them. Things are hopefuly a lot better then they were a week or two ago, and will continue to improve. In the meantime, the vote fary (the nightly process that gives people their allotment of votes) hasn't been running. I'm writing this node primarly to let you know that it isn't just you. Everybody has been affected equaly. (Well, everybody has recived no votes. The number of votes they didn't recive varies by level, just as it normaly does.)

Perlmonks uses three servers, generously donated to us by the good folks at Pair Networks. The database server has recently been running slow. It's recived a new version of FreeBSD, a new compile of MySQL, and another 512MB of RAM, all compliments pair. Last night, the DB server was down for a little while, around when the vote fary was supposted to run. Everybody simply assumed that that was the reason she didn't run, and that it would fix itself.

A god will probably run the vote fary manualy, and fix the problem, shortly. In the meantime, <font size="large" style="friendly">DON'T PANIC</font>. Everything is under control.

Warning: Unless otherwise stated, code is untested. Do not use without understanding. Code is posted in the hopes it is useful, but without warranty. All copyrights are relinquished into the public domain unless otherwise stated. I am not an angel. I am capable of error, and err on a fairly regular basis. If I made a mistake, please let me know (such as by replying to this node).

  • Comment on PM News: Vote fary missing, and other stories...

Replies are listed 'Best First'.
Re: PM News: Vote fary missing, and other stories... (corrections)
by tye (Sage) on Feb 05, 2003 at 17:11 UTC

    The DB server did get a FreeBSD update.

    The DB server has been unacceptably slow during much of each day because it is running out of RAM and CPU. This is probably primarilly due to increased popularity of the site.

    Pair has approved doubling our RAM from 512MB to 1GB. This could make a huge difference. Though it is still possible that we would not have sufficient CPU power for the busy times of the day. We'll see.

    The bad news is that our current DB server hardware can't accomodate more RAM. The good news is that pair will be replacing the computer so we'll get more RAM and probably at least a little bit faster of a CPU. No schedule on this that I've seen yet but pair does good work and I expect one will be coming when the needed resources get lined up.

    Pair has been working on installing a new build of MySQL (my guess is that it could give us an increase of about 10% in performance since we've mostly worked around the problems with MySQL on FreeBSD but it could be no improvement or it could be a bit more than 10%). Although the new build worked on a test server, it wouldn't stay running on PM's production DB server. So we are still running the same MySQL build.

    [ Update: That would (possibly) be a 10% increase in capacity which could mean pages loading much faster when site load is in that range; not a 10% increase in page load speed (which would be a disappointing improvement). ]

    It appears that the work to try the new build of MySQL broke the process that monitors and automatically fixes the IPC used for intra-computer database work. When I investigate the first vote fairy failure, I found the IPC broken and pair corrected it (restarted MySQL) and restarted the monitor daemon. I've notified pair that the monitor has failed again so they're probably already looking into why it failed and fixing it.

    In the mean time, I'll probably update the vote fairy to use a regular network connection to get the DB since that access method hasn't been failing.

    I'll look into running the vote fairy this morning. However, it usually runs during a much slower time and so it might slow down PM too much.

    Early in January, the site got extremely slow (probably just due to popularity). pair greatly improved the situation by removing daemons from the DB server that were using up RAM. One unfortunate side effect of this is that jcwren's PM stats database is no longer getting updates. These updates will probably remain disabled until the DB server gets replaced.

    Note that we can't easily split the DB server load between multiple computers. Also, there are a lot of possible software changes that could make a big dent in DB load. As always, not being able find people with both the time and the knowledge required to make these changes is what has delayed getting these implemented. There are also already several other threads on aspects of these problems.

                    - tye

      It's probably the 'house' boggles' fault---after all you insulted his cousins by misspelling 'fairy'. A small cup of brandy left in the rafters should fix things right up! And don't use the cheap stuff, that's what got us in trouble the last time...


      "Never try to teach a pig to wastes your time and it annoys the pig."
      If jcwren's stats aren't being updated, you can view your XP ranking at my alternate stats listing page. It isn't pretty, but I guess it must be pulling data in a different manner, since I haven't noticed any missing data this month.


Re: PM News: Vote fary missing, and other stories...
by AcidHawk (Vicar) on Feb 05, 2003 at 10:17 UTC

    Thanks for the heads up... If I had a vote it would be yours..;-)

    Of all the things I've lost in my life, its my mind I miss the most.
Re: PM News: Vote fary missing, and other stories...
by crenz (Priest) on Feb 05, 2003 at 09:40 UTC

    all compliments pair

    Those guys drive red purple Ferraris, so they can afford it. ;-)


    Sorry. Insider joke.