AJAX to reduce server load

Looking back over previous nodes (Perlmonks AJAX Voting, AJAX replacement for voting buttons, AJAX-based Perlmonks.com?), I've seen that in the past there's been resistance to the idea of using AJAX on the site. Objections usually amount to the following (and approximately in this order):

We don't need it - the desire to use AJAX derives from a misguided wish to use the newest fad.
It will require JavaScript to be enabled to use the site - and the gods deem this unconscionable.
It will increase server load - Because we'll be hitting the server more often(?)

Before discussing this last point, I have to address the first two. So, first of all, this write-up is not about using AJAX because it's cool, or because it'd make the site seem more modern, or even to improve usability (although IMHO that might be a good reason). I'd like to discuss it here only because it might be possible to use it to reduce server load.

Secondly, of course AJAX in itself requires JavaScript. However, there isn't any reason the site can't degrade gracefully (to its current behavior) in the event JavaScript is disabled or otherwise not available. That is, in fact, best practice for any use of JavaScript, so I see this as a non-issue, given that things are done properly with respect to users who don't use JavaScript.

So, that having been said, the main reason previous AJAX additions could increase server load is simply because the server doesn't have support for them. That is, most of these AJAX requests are going to force a complete page re-load in the background, where it's not even visible to the user. But if the server were set up to expect and handle AJAX requests differently, it could be streamlined to simply do the minimal amount of work necessary for this request, and only send a small amount of data back as acknowledgement, rather than rendering the user's entire site view. I am not a pmdev by any means, but isn't this at least possible?

Here are some areas which I think could benefit from the use of AJAX, if the server could handle them in a streamlined fashion (which is to say, by doing less processing and returning less data than a full page reload):

Voting - Obviously, this has come up before. I think that in particular, if the server were to only send back raw data on the reputation of the nodes on which you've voted, and updated XP information, it could be doing a lot less work than it is now to process votes. After all, when you vote that's the only new information you receive (besides updates to your other nodelets), and yet the server still renders the entire site view. The simplest solution wouldn't even involve an interface change, simply overriding the listener on the "vote!" button to do the work, and if JS isn't there, it works as before.
Drilling deeper into threads - Currently I have my display depth set to 10, because I hate to have to click on deeper nodes to view threads in depth. It should be possible to drill down into threads by retrieving only the node which you want to see, and inserting it into your current view, rather than rendering the entire page again. If this were available, I could decrease my depth to 2 and only drill into threads I want to read.
Editing Previews - Again, simply render the preview and return it, rather than a full page load.
Chatterbox - As far as I can tell, support for this is already there, since the chatterbox sidebar does pretty much the same thing as the free nodelet hack posted earlier. The good thing here is that the sidebar already renders a limited view, rather than a full page load, which means that (hopefully) the AJAX-enabled CB is not bogging down the server as much as clicking "talk" to refresh would do. Of course, it happens more often, but the sidebar does that too. However, since I don't know the server-side details, maybe there is room for improvement here.
New Answer Notification - Okay, I don't know if this would help or not as it depends on how people behave. I know that personally I like to re-load the original thread after I write a response to see if somebody has beaten me to the same answer, to avoid giving redundant (or perhaps wrong!) information. Rather than re-loading the entire original thread in a full site view, again it should be possible for me to check whether new responses exist (a trivial amount of data), and then if they do only return the content of those notes (a smaller amount of data than a full load).

In the end all of these suggestions potential to reduce site load depends on the answer to this question: Which requires the most work for the server, receiving and authorizing incoming requests, or retrieving and rendering all the nodes from the database necessary for a pull-page view?. If it's the former, then these suggestions are a bad idea, since they all require authorization. If it's the latter, then these suggestions could help, by reducing the amount of node rendering the server has to do for these common activities.

Let me conclude by reiterating: This is just a proposal to get feedback, I am not a pmdev, and I don't think we should use AJAX just because it's cool. I'd just like to discuss whether it could actually help with the existing concern of server load or not.

Comment on AJAX to reduce server load

Replies are listed 'Best First'.
Re: AJAX to reduce server load by grinder (Bishop) on Mar 20, 2009 at 14:19 UTC
[lots of arguments in favour of AJAX snipped] You know, I fully agree with you. I think a lot of resistance is due to the perception of sunk investments and that AJAX is a disruptive technology from the point of view of PM infrastructure. PM wasn't build with asynchronicity in mind and adapting the existing code to use AJAX throughout would be a herculean task... and even then you still wouldn't get the same payback as from an architecture which was designed to take it into account in the first place. Yes, wanting to comment on a thread should pull down a DOM component to add the necessary editing widgets to the page. No need to re-render all those time-soaking nodelets. Yes, voting should just push back the decisions and retrieve the node reps. Yes, the chatterbox could do its own refresh. but... Such enhancements would require significant time and effort to build; you'd be fighting the codebase every inch of the way. But God, it would be nice to drag this site into the 21st century, if only from a Perl Public Relations point of view. I spoke with a C# programmer some time ago and when I mentioned Perlmonks... he knew it. He sniggered. • another intruder with the mooring in the heart of the Perl	[reply]
Re^2: AJAX to reduce server load by mr_mischief (Monsignor) on Mar 23, 2009 at 14:04 UTC
It's easy to have a newer-looking site when your implementation language is newer than the first few tens of thousands of posts on the site to which you're comparing it.	[reply]
Re: AJAX to reduce server load by Fletch (Bishop) on Mar 20, 2009 at 13:08 UTC
I really like the editing preview suggestion and I'd think that'd be trivial to implement since it really doesn't need much back-end interaction (maybe resolving `[id://]` links to node names at the most; authentication (probably) wouldn't even be needed). Of course one could probably whip up something on their own that gets 90% there (sans the node titles) with a free nodelet hack . . . hrmmm. The cake is a lie. The cake is a lie. The cake is a lie.	[reply] [d/l]
Re: AJAX to reduce server load by ruzam (Curate) on Mar 20, 2009 at 14:41 UTC
JavaScript (and consequently AJAX) is a tool that can either be used to enhance a website, or abused to crud it up. As Perl culture, I think it we have a history of being open to all the tools at our disposal. Those dismissing AJAX out of hand because it's a 'fad' or requires JavaScript IMO are short sighted. As to whether pm could use a little AJAX, I have no opinion, but the areas you suggested for enhancement don't strike me as providing any solid benefit. Server load is made up of more than just data retrieval and page creation. There's also session management and data verification which must occur regardless of how many bytes are returned. Voting for example, still needs to verify the user session, and validate the node being voted on and retrieve the new vote count. This could add up to nearly the equivalent server load of the existing method or rendering the entire site view. Then you'd have to make additional AJAX calls to update all the other bits that come with the entire site view (cb, messages, xp). So now you've split one request, with one overhead, into several requests, each adding their own overhead. Now factor in that you can vote for several nodes in a single request without AJAX, but you have to make a new request for each vote with AJAX and you've multiplied the server load many times over yet again. The most obvious reason to use AJAX is to enhance the user experience with a seamless 'non-refreshing' website and I think this should be the fundamental point. Arguing for AJAX based on sever load is a red herring IMO.	[reply]
Re^2: AJAX to reduce server load by bellaire (Hermit) on Mar 20, 2009 at 14:51 UTC
I can understand that authorization takes time, however I don't think you've really given the ideas I put forth a fair shake. Particularly when you say: Then you'd have to make additional AJAX calls to update all the other bits that come with the entire site view (cb, messages, xp). So now you've split one request, with one overhead, into several requests, each adding their own overhead. Now factor in that you can vote for several nodes in a single request without AJAX, but you have to make a new request for each vote with AJAX and you've multiplied the server load many times over yet again. With respect, there is no reason you'd "have" to do things that way. That would be a very poor implementation. Obviously if you are going to send several AJAX requests for something that currently takes one request, you face diminishing returns. However, in my write-up, I specifically said that the voting AJAX could work just like the current interface, overriding the "Vote" button itself. This means that we have merely replaced a full page load (on submitting the vote) with a single AJAX request (on hitting the "vote" button). The entire point of the post was to identify situations which currently require a full page load and replacing those common activities with a simplified (single) AJAX request instead. This would not require increasing the number of trips to the server, but merely doing some of those trips via AJAX rather than reloading the page. I can accept that such a replacement would provide little benefit if the work done for an AJAX request is almost the same as it is for a page load. However whether that is indeed the case is a question for the pmdevs.	[reply]
Re^3: AJAX to reduce server load by ruzam (Curate) on Mar 20, 2009 at 15:54 UTC
Sorry, I mis-understood your vote example. But on the same train, if you're proposing (for example) that voting would work the same, without the full page load, then you've added nothing to the user experience, and 'maybe' only reduced the sever load a wee bit. I say a wee bit, because voting probably counts for a very small fraction of the site traffic. I'm sure the load is the result of casual browsers, not hard core voters (numbers anyone?). With AJAX voting as you've suggested, there's still the question of cb, xp vote counts and 'new' replies to the node you're currently viewing. These are updated with the full site request. The AJAX voting would have to either request and return this information as well (making it nearly as intensive request as a full page) or resort to additional requests to update the information (again increasing load). IMO your vote scenario is at least as load intensive if not more than the full site request. Of course, if you're talking about voting in the voting booth (not on nodes), then I'm trying to compare apples and oranges ;) I think the voting booth would be an excellent candidate for mixing AJAX to reduce server load. I also think using AJAX to update individual node votes (as I originally thought you intended) would also be a positive user experience, with a nasty load price. I think the cb could really benefit from AJAX. I'm not a big cb user, but I like to keep my eye on it while I'm reading nodes. I've always found it 'dis-jointed' to see the conversation skip along with each new page request and a real time flow of comments would be sweet. But doing so would replace the occasional page refresh with a constant nattering of AJAX 'pings' that surely will load the server even more. On the other hand, cb doesn't require any authentication to 'view' it, so overhead could be reduced. But I stand by my point that AJAX if used should be for the benefit of the user experience, and arguing that it reduces (or increases) server load is misdirection IMO.	[reply]
Re^4: AJAX to reduce server load by puudeli (Pilgrim) on Mar 20, 2009 at 19:11 UTC
Re^4: AJAX to reduce server load by Your Mother (Archbishop) on Mar 21, 2009 at 00:14 UTC
Re^5: AJAX to reduce server load by ruzam (Curate) on Mar 21, 2009 at 03:07 UTC
Some notes below your chosen depth have not been shown here
Re^2: AJAX to reduce server load by Anonymous Monk on Mar 23, 2009 at 12:41 UTC
Voting for example, still needs to verify the user session, and validate the node being voted on and retrieve the new vote count. This could add up to nearly the equivalent server load of the existing method or rendering the entire site view. Then you'd have to make additional AJAX calls to update all the other bits that come with the entire site view (cb, messages, xp). So now you've split one request, with one overhead, into several requests, each adding their own overhead. Now factor in that you can vote for several nodes in a single request without AJAX, but you have to make a new request for each vote with AJAX and you've multiplied the server load many times over yet again. you don't know that of which you speak. doing a vote update does not require the cb to update. updating the cb doesn't require the votes to update. think man, THINK.	[reply]
Re: AJAX to reduce server load by rhesa (Vicar) on Mar 21, 2009 at 10:44 UTC
Used properly, AJAX can make big improvements to site usability. There is one issue with loading partial content through AJAX that annoys the heck out of me though, and that's the interaction with the browser's back and forward buttons. When using those buttons, I often notice that the AJAX parts get reloaded from the server over and over. Examples of this include browsing source trees on Github and in Trac. Given that issue, I do think AJAX could improve voting, and editing previews, and the chatterbox. For loading and rendering nodes, I'm not so sure.	[reply]
Re: AJAX to reduce server load by eric256 (Parson) on Mar 23, 2009 at 18:11 UTC
I looked into this specificaly with voting a while ago 656735 and found that there was a significant advantage network wise. Its hard to say from that what the advantages would be for processor time etc without someone more familiar with the server. It should also be easy enough to adapt to submit all votes on the page via AJAX at once instead once for each vote. (That code is 2 years old so please don't laugh at me, i havn't looked at it or used it in ages) ___________ Eric Hodges	[reply]
Re: AJAX to reduce server load by SFLEX (Chaplain) on Mar 21, 2009 at 12:18 UTC
I think for voting AJAX could speed up the site(if made right), but for the CB well there are so many CB's for PM that have AJAX like functionality that i don't see a point to make another unless its a personal CB. Spiel auf H�ndinnen.	[reply]
Re: AJAX to reduce server load by Anonymous Monk on Mar 23, 2009 at 14:57 UTC
I know this might sound archaic to most of you, but I use lynx as my main browser, and it doesn't support Javascript (and thus AJAX). I use Lynx because I prefer it to the bloated graphical browsers. It runs in a terminal with the same nice fonts I use when coding (in Vim, another "archaic" tool, I guess). It can run under a GNU screen session. It supports some level of scripting and automation, using simple Perl or shell scripts (and works exquisitely well with the "surfraw" search tools). It can be used exclusively from the keyboard, which is a big plus for me, especially when using a laptop that doesn't have a proper pointing device (not that I much enjoy reaching for the rodent in the first place). It let me bypass all the ads and junk on the web these days and focus on the content. It's also fairly secure, especially when you consider I'm using the version that ships with OpenBSD. I have used this web browser since 1995 (an age when many gopher sites were still active), and feel that it still delivers a web experience that's above and beyond what the others do (the "links" family comes close though). It would be a great pity if perlmonks changed their site in such a way as to make it unusable for anything but the graphical, mouse-intensive, java(script) ladden heavy browsers that are common these days. Some of the concerns I bring up may also apply to WAP devices, though I can't be certain about this since I don't use those myself.	[reply]
Re^2: AJAX to reduce server load by bellaire (Hermit) on Mar 23, 2009 at 15:43 UTC
Absolutely agreed. Let me reiterate that I don't think any JavaScript should be applied which doesn't gracefully degrade to the current functionality if JavaScript is disabled or unavailable.	[reply]