Well, actually, PerlMonks is based on an earlier version of the Everything engine which differed substantially from what you downloaded, even aside from the massive customization we've made to it over the years.

Even so, it's possible that Cache::Memcached::Fast could be worth using here. (Unfortunately I'm not quite qualified to comment beyond that.)

At least, we still use a/the version table to cache versions of nodes, but we also have other caches for long-running queries or queries that only need to change their results rarely (Best Nodes, Chatterbox contents rendered as HTML, ...). For these, it might make sense to store/retrieve them from a cache separate to the database.

I'm a tiny bit reluctant to introduce another separate component to the site, and I have no experience with how the memcached clients behave if no server is available. But as I found that memcached is available for Win32 and has a pure Perl implementation, I'll look into using it, as it could shift some load away from MySQL.

and I have no experience with how the memcached clients behave if no server is available

no problem. if there is no memcached node available, then you use the database. the frontend module will just return undef for the lookup.

Why hesitate? I've been always wondering why the biggest international perl forum (I know) is such a slow website. I thought, there must be a lot of experienced web developers hanging around. Introducing memcached sure is another component, but it's a cache - if there is nothing in the cache, it returns undef and you create what you need from the database. where's the problem?

Yeah, memcached would very likely provide a noticeable speed boost and reduce memory usage if its use were properly implemented here. It wouldn't eliminate the per-process node cache, just make it faster and smaller. And I probably wouldn't eliminate the version table...

I'm sad to see that memcached has still not provided a way to avoid the race condition that I pointed out several years ago (and that is quite simple to avoid). And you avoid it by having a version number (usually one that is sequenced when you route your updates through the database).

memcached provides ways to avoid other race conditions by supporting increment/decrement, append/prepend, and 'cas' operations (which I believe are all new since I looked at it last).

But the original race condition still exists. Imagine one web server gets bogged down and the following sequence of events happen between web server X and web server Y:

X: request arrives when Bob downvotes Alice's node, N
X: read version 1 of N from memcached, N1
X: decrement 'reputation' field in N1, producing N2
X: flush N2 changes to DB (update ... rep=rep-1 ...)
X: re-read N from DB, yielding N2 again
X: the slowness of this web server matters at this point

Y: request arrives when Alice updates her node, N (having noticed a typo that makes her node appear extremely rude)
Y: read version 1 of N from memcached, N1
Y: apply update to node text, producing N3
Y: flush N3 changes to DB (update ... doctext='...' ...)
Y: re-read N from DB, yeilding N4 (includes both the text update and the reputation decrease)
Y: flush N to memcached, storing N4

X: flush N to memcached, storing N2 (Oops!)

Y: redirect to prevent POST being re-applied
Y: get request to show N to Alice as response to her update
Y: read N from memcached, getting N2
Y: display N2 to Alice, who wonders where her update went!

Alice can refresh and refresh and she still won't see her update. When Carl looks at N, he also gets shown the rude version. Carl is offended by Alice's extreme rudeness and finally downvotes her node, putting N5 into the DB and memcached (N5 includes Alice's update and both decrements to the reputation).

In response to Carl's downvote, he is shown N5 and realizes Alice's rudeness was just a typo.

Meanwhile, Alice has written a PMD node wondering how her update could be ignored like that. This makes her look foolish because her update is plainly visible now (even though it wasn't visible for a long time after she made it).

Why do the memcached developers hate Alice so much?

Yes, there are several other race conditions in this scenario. I've talked about them several times before and even described how I mitigate them. Note that the first block of steps for Y are just part of a single web request and so can happen very quickly so X need only be slightly bogged down to make this possible.

Some observers may have noticed that some of the steps described are not actually how PerlMonks currently works (in particular, "rep=rep-1" and the redirect). The deviations from current reality just make the race consequences different, not less of a problem. For example, Alice might see her update yet nobody else does but later she refreshes and her update is gone, only to reappear much later when somebody votes on her node.

This added race condition might not be enough to prevent memcached from being used at PerlMonks, but it certainly doesn't help motivate adoption of it. The real blocker will likely be the difficulty in getting major updates deployed (my node cache re-write still isn't deployed, darnit!).

Thanks for the recommendation!

Oh, the way to avoid the race is to allow a memcached 'set' operation to include a "version string". A 'set' operation using a version string lt the current version string (using Perl's concept of lt) just gets ignored as a stale update. If you want to use numeric versions for memcached objects, then just be sure to zero-pad them to an appropriate, fixed length.

I've been asking on where the problems are several times. It would be useful to run something like munin and look at a statistics on which pages are retrieved most. It's better than making a wild guess.

I'm using caching of data structures for my own forum software. It really helps a lot. I have some experience in scaling a large website. I just need an instance with the current software in use and *some* statistics. installing/configuring munin is done in 15 minutes or so.

I've been asking for munin statistics, log files, I wrote a skeleton script which creates a KinoSearch index, for moving the current mysql "like" search to KinoSearch. (Using KinoSearch for my own forum software. It is - of course - so much faster than a mysql like query.)

Obviously the problems are not big enough yet, or I'm talking to the wrong persons.

So yes, memcached or anything would help, but if nobody lets us get to the code/machines to actually do something, it will stay like it is forever... =(

I've been asking on where the problems are several times. It would be useful to run something like munin and look at a statistics on which pages are retrieved most. It's better than making a wild guess.

Well, I wasn't guessing wildly. Page load numbers were even public information until they got broken (by a mysql update, I think -- I don't have those details swapped in at the moment).

And I've posted public nodes about the problems that have been identified, how many of them were fixed, and how they get rebroken. It is too bad that you seem completely unaware of them. *shrug*

Update: Sorry, I'm sure that sounds harsh. Let me echo Corion's best reply. Your help and support is sincerely appreciated! Very little in PerlMonks admin is instantaneous, far from it. I wouldn't rate memcached as the first priority but I'm also convinced it would be an improvement (except for the race condition which might hardly ever be noticed or might be quite an annoyance that other users of memcached don't run into).

- tye        

except for the race condition which might hardly ever be noticed or might be quite an annoyance that other users of memcached don't run into

I didn't get that race condition anyway. Especially the "flush to memcached".
What i'm doing in my forum at the moment is storing a whole thread in memcached as a data structure. Expiry is set to 3 minutes (could also be more, just because I'm storing also user info and signatures, it's set to 3 minutes, so that user info changes get updated faster).
Now, whenever somebody posts a new node or updates their node, the cache entry is explicitly expired immediately. Database transaction, after that delete thread cache entry, then redirect the user to the updated thread or node. And the request for the thread then reads from the database and creates the cache entry again.
Your use of memcached seems a bit different, if I understood that correctly? So the wrong flush to memcached wouldn't happen in my case.


edit: yes, I think that's the point - in the second block of server X you are flushing N2 to memcached, but the current version in the databse is N4. If creating the cache entry only when fetching a thread the RC shouldn't be there.

edit 2:
or maybe I can still create an RC:


X: reads thread with node N1
X: votes N1 -> N2, delete cache entry
X: redirect after POST and reads N2 from DB...

Y: author updates node from N2 to N3
Y: delete cache entry, if exists
Y: redirect after POST and reads N3 from DB
Y: cache N3 to memcached

X: ... cache N2 to memcached
[download]

But that version must have a whole transaction and redirect and memcached operation in between that X blocks. of course, it's still possible, but extremely improbable. Could maybe be prevented by putting the read and creating the cache entry into a transaction with read lock?

installing/configuring munin is done in 15 minutes or so.

Despite the Munin site list of prerequisites never mentioning anything about permissions, that I noticed, the "INSTALL" file says:

Create the user "munin" and the group "munin"

I no longer even have access to run /usr/bin/top. I certainly can't create users, even given the mythical 15 minutes (took that long just to find out whether 'root' was a requirement).

P.S. I believe this thread is the first I have ever heard of Munin (much less repeated requests to have it installed).

- tye        

Well, too bad that you've never heard of it. In most of my past jobs it was used as a monitoring tool (besides nagios). It's really great (and writing own plugins in perl is fun), and I can only say on debian I do "aptitude install munin munin-node munin-plugins-extra" and then activate the plugins for mysql und apache and I'm done.
Of course if you're totally new to munin then it might take you longer than the 15 minutes. And yes, you have to be root (I don't know if it's possible to run it as a normal user, some of the values it queries might only be available to root). So, no munin, ok. I mean you don't have to. You could use the tool you like best =)
Still, if you need help with configuring munin, just ask. I'm also in irc.perl.org

Here's an interesting presentation from OSCON 2009, essentially a bit of memcache marketing:
Controlling the Addiction: Best Practices for Scaling With Memcached