Over at E2, we are about to take what is going to be an very incompatible change and forever move away from Perlmonks: we're going to be getting rid of the $VARS hash, and replacing it with a $DB->getNodeParam($NODE, "paramname") type setup where we have a large, indexed key-value store for each node. We'll be caching the contents of the vars in the nodeCache, but the goal is to not have to rewrite $VARS every time, and to be able to expand the code so that any type of node can have a key/value parameter.

The goal is to move some non-index-needing, non-primary columns over to it, and be able to add arbitrary metadata to any type of node, without having to muck with types and database schemas. If you guys are interested in syncing up, drop me an email or a message here, and we'll try and figure out what our engine deltas look like. In ecore terms, it is like every node would virtually join on setting.

I'd like to continue to be on the same rough engine branch, and be able to share performance improvements with you guys, but it's going to require a bit of engineering work.

Also, we've had some great success moving to AWS, so if you're thinking about picking up stakes and moving anywhere, please let me know and i'll have happy to share my recipes and practices getting E2 up and cooking on a modern web platform. Having S3 and RDS available has been a huge boon for our performance. I don't know what the state of the site is now that Perl foundation has it.

Update: The below has been done. You may disregard. :-)

If you take a look at the code of each of the sections, you'll see a fairly common pattern. SoPW is the canonical case; it looks like this:

[{get_sitedoclet}]
[{votehead}]
[{newlistapproved:perlquestion,perlquestion approved linktype,User Que
+stions,10,navbaron,showunapproved}]
[{votefoot}]
[{newmoderatelist:perlquestion,perlquestion approved linktype,Unapprov
+ed Questions}]
[{addnewform:perlquestion,Add your question,Your question:}]
[download]

Examining newlistapproved and newmoderatelist, we find that the latter is logically a part of the former, but it seems to have been split out merely in order to "jump over" a call to votefoot.

I propose merging the code of newmoderatelist back into newlistapproved, and add an option parameter which would tell newlistapproved whether to call votefoot at the appropriate point. There is already an option parameter, showunapproved, which controls whether to show the unapproved list.

In order to grasp all the ramifications across the site, I prepared the following table, which summarizes the code of each place newlistapproved is currently used (excluding test nodes and such).

Table is presented in a reply below.

Based on this data, I believe this change would be safe and straightforward to make. The only real oddball we'd have to worry about is Editor Requests, but I think we could handle that.

I think this change will make the code simpler, but perhaps more importantly, more efficient. The overhead cost of calling an htmlcode is substantial. It includes the cost of (a) retrieving the node from the database, and (b) executing a perl eval.

• vagrant up, environment similar to E2, based on squeeze and perl 5.10
• Bootstrap, however lots of server errors on the page (go to http://localhost:8888/?node=Login once it is up).

• Pieces to generate cachedgates.html
• A few needed dependency modules need to be added to the chef recipes. I saw one, but forgot to jot it down. I'll add it once we work through a few of the errors.
• The pmscrape.pl script stupidly scrapes Node Lister for every category, because it's a dumb comparion. Also, my username is hardcoded in.

Some of you may or may not remember me, but I'm Jay Bonci, a longtime perlmonks semi-lurker, and the new owner of Everything2, the PM sister-site. Even though it diverged a long time ago, I'm hoping that things are still similar enough I'm back around actively developing and improving the core engine underneath E2 with a huge push to clean up a lot of old practices and bring the code forward to the modern era. I'd love to see if Perlmonks can get some use out of the tools that we're developing.

I've got a somewhat active repository going at github with our tools. Looking ahead to where I'd like to be:

• The development community should be developing inside of a bootstrapped sandbox machine running inside of a Vagrant virtual machine. This allows us to develop Chef recipes for box deployment as well
• Nodes are stored in git(hub) as XML with ecoretool.pl export, and are instantiated with ecoretool.pl import (or bootstrap). This allows for a sanitized environment and source control of the underlying nodes
• We're working towards compilation of the underlying htmlcodes, containers, htmlpages, opcodes, superdocs, etc that run the site and I've developed an execution environment using some symbol table muckery to make it think that it's running inside of Everything::HTML. This is going to reduce our apache footprint and our database load pretty significantly. It also allows for Devel::NYTProf to not get lost inside of evals

I'd be greatly interested in working with the PM staff to see what tools you can can lift or vice versa. Check out the github repository and message me here or over email on my homenode. The goal is to rapidly improve our body of in-database software and to get out of the performance bottlenecks that are increasing how much it costs to run the site (or conversely, how much capacity we get out of our current hardware).

Idea: Let the permission denied page contain information on the node you're not allowed to see.

In particular, I want the page to display the sitedoclet associated with the denied node, if any. This will be useful to SiteDocClan as it will facilitate the maintenance of sitedoclets associated with nodes they're not allowed to "run". A well-written sitedoclet might also allay some of the confusion to which newbies are heir, when they hit a forbidden page.

Technical Implementation

The code of permission denied needs to know which node the user requested and was rejected. Currently, that information is lost (afaict) as soon as the engine decides to redirect to permission denied. Therefore, I propose the following patch in order to preserve that info:

In Everything/HTML.pm, insert the following line before line 1408:

  $HTMLVARS{requested_node} = $NODE;
[download]

When this has been done, the Permission Denied - (patch) can be applied.

Update: Done!

You know how being /borged prevents one from talking in the cb but doesn't prohibit one from posting... I'm thinking we maybe could use an op of the opposite behavior as well — preventing one from posting (or editing their already-posted nodes), but not stopping them from talking in the cb. (I'd call it the "brig".) Seems to me we could have used this recently. Instead, an all-or-nothing solution was used, and that in turn drove the aggrieved monk to create a new account. What say ye?

Update: Please see the update at the bottom. I think I was mistaken in assessing my observation.

It's probably been this way for awhile, but I've been wondering why when a person gains XP the notification message in the XP Nodelet sometimes may report the new XP gain several times in a row before the message stops showing up.

Here is the relevant code, I think:

 1: my( $shownumbers )= @_;
   2: # send FALSE if you want people to see gain/loss w/o exact numbe
+rs
   3: 
   4: return ""
   5:     if  getId($USER) eq $HTMLVARS{default_user};
   6: 
   7: my $curexp= $USER->{experience};
   8: my $oldexp= $VARS->{oldexp};
   9: $oldexp= $curexp
  10:     if  ! defined $oldexp;
  11: my $difexp= $curexp - $oldexp;
  12: $VARS->{oldexp}= $curexp;
  13: return ""
  14:     if  0 == $difexp;
  ....... lots more below dealing with quips.
[download]

It seems that the desired behavior is to have a new XP gain reported exactly one time, not several. I can't prove that it's not just a matter of being upvoted several times, but my feeling that the upvote clusers don't fit my posting habits. For example, maybe I've gone several days between new posts. When I come back to PerlMonks, I find 6xp added. Then upon hitting, "Newest Nodes" or "Talk", I get the same six-point report. It's unlikely in that event that I really gained 6 over the course of a couple days, and then six in the matter of a few seconds again..

I don't want to put out a theory as to why it's happening in the absence of solid evidence, but it seems that $VARS->{oldexp}= $curexp; is taking several reloads to propagate to all the servers.

I guess I would be looking for confirmation that (1) There is a problem, ie, the behavior is as I describe: That the first two reloads (even via 'talk button') sometimes fail to clear the New XP message by setting $VARS{oldxp}=$curxp, and (2) That it's supposed to work as I expect: The report is supposed to show only once. A third question would be what possibilities exist as to why it's happening.

Update: Now that I finally gather the courage to suggest there might be a problem I find it impossible to duplicate the behavior I thought was happening. So perhaps $VARS->{oldexp}= $curexp; is doing exactly what it ought to, and I've been incorrect in my casual observation of the anecdotal evidence I thought was supporting my hunch. ;)

Thinking about how to blur the distinction between nodelets and regular nodes, so that you could, for example — theoretically — embed a regular node into your page like a nodelet, or — more practically — view a nodelet as a standalone ("regular") node.

Nodelets are special in the following ways:

• they have a designated "container", which is essentially HTML template code which gets wrapped around the nodelet's contents at display time;
• they have the concept of periodic updating, with caching of contents in between updates.
• they can be selectable for inclusion in your list of displayed nodelets.

Aside from that, they are essentially superdocs: documents which can contain code for performing various functions. There is no technical reason why a nodelet could not be viewed standalone, just like any other superdoc.

Therefore, as a proof of concept, I created nodelet view page, a displaypage which does exactly that. To try it, view any nodelet and add ;displaytype=view to the URL. For example, your XP Nodelet.

It works. However, there are obviously some caveats. For example, viewing your Free Nodelet in this way is likely to cause mayhem if you have any javascript going on in it.

Now, if we can get this code to be ready for prime time, I would recommend making this the default view for nodelets, and migrating the current default view into the viewcode htmlpage. That is:

1. rename nodelet display page as nodelet viewcode page, then
2. rename nodelet view page as nodelet display page.

Then a pmdev simply clicks on the "code" link which will appear at the top of the nodelet display page to get to a point where its code can be examined/patched.

One further kewlosity: add /bare/ to the path part of the URL to get a nifty little ticker-like form (html, not xml) of any given nodelet. Example: your XP Nodelet. (It's not really a ticker; it doesn't auto-refresh.)

I also have a patch (expandfreenodelet - (patch)) for embedding any given nodelet's contents into your Free Nodelet. I'd like to know if this might have potential problems. It seems like a cool idea to me.

How about md5 crypts instead of password in the cookie? That would allow passwords > 8 chars (with a user table change).

md5 may be too CPU expensive; needs to be tested.

An md5 crypt certainly takes more time than a des crypt: one some machine the md5 crypt seems to take 5e-4 second, the des takes 1.3e-5 seconds.

(comments about $3$/NT-hash)

Instead of comparing hashed password in cookie to hash of clear password in database, store the hashed password in the database and the non-salt part of it in the cookie; authenticate cookies via string compare.

Later, unhashed password will be eliminated.

update user edit page to require the previous password in order to change the password

have a real "password reset e-mail" feature

Following up on previous thread Pmdev documentation...

I've been cogitating a bunch upon this idea of using sitedoclets to document code, i.e. infrastructure nodes.

Currently, sitedoclets can be attached to infrastructure nodes of the following types:

• htmlcode
• usergroup
• setting
• accessrule

Anyway, I've come up with what I think is a pretty good idea, so I'd like to run it by you all, see what you think.

My idea essentially is this: have a new nodetype, devdoclet, which would be used to document the purpose and usage of infrastructure nodes. In particular, it would be a great place to note the status of nodes such as dead, live, experimental, and not-yet-live htmlcodes .

devdoclet is like sitedoclet in most ways, except that,

1. It is owned (creatable, editable) by pmdev rather than SiteDocClan, and would not be intended for general public consumption.
2. More importantly — there is an explicit policy which supports bi-directional links, implicitly by title, between an infrastructure node and its devdoclet.

That is, the doclet for (say) parselinksinchatter is necessarily parselinksinchatter devdoclet. (It would therefore be slightly different from the sitedoclet situation, where this nomenclature is conventional but not universal nor enforced.)

This enables certain very convenient things — most obviously, that a pmdevil can navigate from a devdoclet to its associated code node simply by stripping off the " devdoclet" part of the title. (Of course, we'd automate this for you by means of a link in your pmdev nodelet.)

I also envision a structure, perhaps somewhat like the sitefaqlet/faqlist nested-listy thing, for knitting all the devdoclets into a whole "site infrastructure document"... as touched on in the earlier thread. And note that it would also be possible to make devdoclets which are not linked to specific infrastructure nodes; these could be used to document overarching concepts and like such as.

Thanks...

Update:

2018-12-18

The new nodetype devdoclet and the associated htmlcode showdevdoclet have been created.

In the new regime, sitedoclets are used for storing content which is to be embedded/included in some other displayable object (such as a superdoc or nodelet); and devdoclets are for writeups which serve to document infrastructure nodes, such as htmlcodes and htmlpages. Only pmdevils will have the ability to read/write the latter. sitedoclets remain the bailiwick of SiteDocClan.