Editing and display of nodetypes

The sitedoc infrastructure inherits most of its htmlpages from node,document,doclist, and docstring. There are only three sitedoc specific htmlpages (sitefaqlet display page, sitedoclet display page, and sitefaqlet edit page). Thankfully docstring and doclist have no hard coding and only minor changes need to be made to the faq/doc system:

1. Add nodetypes, setting creator=deleter=updater=pmdev, reader=cabal:
   • pmdvfaqlet: inherits from sitefaqlet
   • pmdvdoclet: inherits from sitedoclet
   • pmdvlist: inherits from doclist
   • pmdvstring: inherits from docstring


2. get_docball_data (htmlcode) - add node.

   This node records in one place all the information currently scattered about for documentation sets. I've only used it in code for new nodes and existing nodes that could use it without restructuring. Some nodes have hard coded data so scattered that it wasn't possible to use. We could go back and revise those nodes to use it as well at some point. Placing this data in code, even in a single code location isn't ideal. It would be rather easy to mess up the data. Ideally, this should be in the database or in settings where it can be edited more easily per-docball.

      # !!!HARDCODED DATA!!! - common data with [rebless]
      # Note: for ease of use hash key names are the same as
      # the section id used by Update Master Faqlist, if
      # one exists. (Note: pmdv chosen rather than pmd to
      # avoid confusion with [PMD] (Perl Monks Discussion)
      #  
      #    faq    SiteDocClan documentation
      #    tut    Tutorials 
      #    pmdv   pmdev documentation
      #    cqa    Categorized QandA
      #
      # array elements in $hNodetypes values:
      #    [0] = type label for nodelet links
      #    [1] = id of master list for nodetype
      #    [2] = 1 if rendered using get_sitedoclet
      #          0 if not
   
      my $hAllDocballs
        = { faq
            => { masterlist  => 481919
               , altview     => "SDC View"
               , RJEfilter   => 'SDC'
               , usergroup   => 'SiteDocClan'
               , listtype    => 'faqlist'
               , stringtype  => 'faqstring'
               , reblessable => [qw(sitedoclet sitefaqlet
                                    alphafaqlet)]
               , nodetypes 
                 => { sitefaqlet => ['faq',1, 48286]
                    , sitedoclet => ['doc',1, 482316, 1]
                    , alphafaqlet => ['alpha',1,481721]
                    , faqlist => ['faqlist',0,481919]
                    , faqstring => ['faqstring']
                    }
               }
          , tut
            => { masterlist  => 743094
               , altview     => 'Pedagogue View'
               , RJEfilter   => 'Ped'
               , usergroup   => 'Pedagogues'
               , listtype    => 'tutlist'
               , stringtype  => 'tutstring'
               , reblessable => []
               , nodetypes
                 => { tutlist => ['list', 743094]
                    , tutstring => ['string', 743096]
                    , perltutorial => ['tut', 743095]
                    }
               }
          , cqa
            => { masterlist  => undef
               , altview     => 'QandAEditors View'
               , usergroup   => 'QandAEditors'
               , listtype    => 'catqalist'
               , stringtype  => 'catqastring'
               , reblessable => []
               , nodetypes
                 => { catqalist => ['list', undef]
                    , catqastring => ['string', undef]
                    , 'categorized question' => ['q',undef]
                    , 'categorized answer' => ['a',undef]
                    }
               }
          , pmdv
            => { masterlist  => ????
               , altview     => "PDM View"
               , usergroup   => 'pmdev'
               , listtype    => 'pmdvlist'
               , stringtype  => 'pmdvstring'
               , reblessable => [qw(pmdvfaqlet,pmdvdoclet)]
               , nodetypes
                 => { pmdvfaqlet => ['faq',1, ????]
                    , pmdvdoclet => ['doc',1, ????, 1]
                    , pmdvlist   => ['list', ????]
                    , pmdvstring => ['string', ????]
                    }
               }
          };
   
          my $hBogus 
             = { masterlist => undef
               , altview => "Abnormal View"
               , usergroup => undef
               , listtype  => undef
               , stringtype => undef
               , reblessable => undef
                 # empty nodetype hash flags this as bogus
               , nodetypes => {}
               }
          };
   
      # If a user group is the assigned editor for more 
      # than one docball, then we need to find the primary
      # one when $id_type = 'usergroup'.  This hash keeps
      # track of each group's primary docball.
      
      my $hUsergroupDocballs
       = { SiteDocClan  => 'faq'
         , pmdev        => 'pmdv'
         , Pedagogues   => 'tut'
         , QandAEditors => 'cqa'
         };
   
      # END (hardcoded data)
   
      # NOTE: if docballs were set up as a nodetype, then
      # $keytype 'node' should also be added
      #
      # $docball_id  id of docball - depends on $keytype
      # $id_type     type of identifier
      #              may be one of the following strings:
      #                  list - id is list nodetype
      #                  section - id is master list section
      #                  usergroup - id is user group.
      #                    primary docball for user group will
      #                    be returned
      #                  all - hash keyed by section
      #                    containing all docballs
      #                  bogus - hash for bogus docball.
      #                    (scalar keys $hBogus->{nodetypes}
      #                     is always 0)
      #              defaults to section
   
      my ($docball_id, $id_type) = @_;
      $id_type = 'section' unless defined($id_type);
   
      # Returning references SHOULD work.
      # Even though the Everything Bible says that htmlcode
      # nodes must return a string, in reality 
      # Everything::HTML::htmlcode returns in scalar context
      # whatever eval returns (unless there is a warning or
      # exception in which case it stringifies the return
      # result and appends the error message).
   
      if ($id_type eq 'all') {
         return $hAllDocballs;
      } elsif ($id_type eq 'section') {
         return $hAllDocballs->{$docball_id}
            if exists($hAllDocballs->{$docball_id});
      } elsif ($id_type eq 'list') {
         # assume that each docball has its own list type
         # and so uniquely identifies a docball
         # otherwise why would we need a separate docball?
         foreach $k (keys %$hAllDocballs) {
            $v = $hAllDocballs->{$k};
            return $v if ($v->{listtype} eq $docball_id);
        }
      } elsif ($id_type eq 'usergroup') {
        my $k = $hUsergroupDocballs->{$id_type};
        return $hAllDocballs->{$docball_id}
            if (defined($k) && exists($hAllDocballs->{$k}));
      }
      return $hBogus; #not a docball
   [download]

3. doclist display page (htmlpage):

      # replace lines 3-8
      my $sType = $NODE->{type}{title};
      my $altlabel = $altview
         ? "Normal View"
         : htmlcode('get_docball_data','',$sType)->{altview});
   [download]

4. render_doclist_group (htmlpage):

      #lines 131-139
      # these hardcode the string 'sitedoc' but for now
      # no change is recommended. SiteDocClan documents
      # have a special feature whereby any node X can have
      # an accompanying document named "X sitedoc". This
      # feature is not needed at present for the pmdv docball
      # - see below "Annotating other documents" for further
      # discussion.
   
      #replace lines 149-151
      if (my $sdl=getNode
          ("$node->{title} sitedoclet",'sitedoclet')
          push @out, htmlcode('get_sitedoclet','',$sdl);
      } elsif ($NODE->{type}{title} eq 'pmdvdoclet') {
          push @out, htmlcode('get_sitedoclet','',$NODE);
      }
   [download]

5. doclist edit page (htmlpage):

      # replace lines 15-20
      my $hDocball = htmlcode('get_docball_data',''
         , $NODE->{type}{title}, 'list');
      htmlcode('groupeditor', 'string'
         , $hDocball->{stringtype});
   [download]

6. sitedoclet display page (htmlpage ): replace node in full

      [%
         # make very sure we pass a hash to avoid
         # defaulting to a sitedoclet. Passing anything
         # other than a hash will trigger the defaulting
         # mechanism of [get_sitedoclet]
   
         $DB->getRef($NODE);$
         htmlcode('get_sitedoclet', $NODE);
      %]
   [download]

7. sitefaqlet edit page (htmlpage): replace node in full

      [%
         # derive updaters group from node
         # Note: this code is also used in [doclist_edit_page]
         # and [docstring_edit_page]
         my $updaters_group = getNodeById($NODE->{type}{updaters_user})->
   +{title};
   
         htmlcode('handle_node_edits', '', $updaters_group);
      %]
   [download]

8. CanPmdevEdit (accessrule): add this rule.

   This rule is needed by handle_node_edits because handle_node_edits calls access rules directly without parameters rather than passing them through isApproved()!

   Note: this does not use get_docball_data because it is dependent on user group and not docball.

      # !!!HARDCODED DATA!!!
      # !!!must be synchronized with group-based rules stored
      # !!!in the database!!!
      my @aNodetypes
         = qw(pmdvfaqlet pmdvdoclet pmdvlist pmdvstring
              pmdevnote);
      # END (hard coded data)
   
      my $node = shift;
      $node ||=$NODE;
      $DB->getRef($node);   #convert to hash if need be
      my $sType = $node->{type}{title};
      return grep {$sType eq $_} @aNodetypes;
   [download]

9. handle_node_edits (htmlcode):
   Note: this does not use get_docball_data because it is dependent on user group and not docball.

      # insert into hash @ lines 16-39:
      pmdev => {
        is_ok_type      => 'CanPmdevEdit',
        not_ok_type_msg => 'Not a Pmdev editable type',
        not_member_msg 
        => 'You're not a pmdevil get away, there is no Perl Illuminati'
      }
      # replace line 143 with:
      } elsif ( $NODE->{type}{title} =~ /^(?:sitedoclet|pmdvdoclet)$/ ) {
   [download]

Master lists

Each document collection (tutorials, site documentation, categorized questions and answers) has its own set of master document lists generated by calling the Update Master Faqlist superdoc with an appropriately set "section" parameter. Current values for this parameter are:

• faq: generates site documentation master lists
• tut: generates tutorial master lists

To extend this infrastructure to include pmdev documentation we need to make the following changes:

1. Create master doclist nodes (the names must match the rules that Update Master Faqlists uses to generate master document list names.)
   • master pmdvlist pmdvlist
   • master pmdvstring pmdvlist
   • master pmdvfaqlet pmdvlist
   • master pmdvdoclet pmdvlist
2. Choose a section label: pmdv
3. Update Master Faqlists:

      # replace lines 5-13 (inside for loop)
      # not quite sure why this is inside a for loop since
      # only one value of $usergroup, etc can be set.
   
      my $section = lc($_);
      my $hDocball = htmlcode('get_docball_data','',$section);
   
      #bogus docball has no nodetypes
      if (scalar keys $hDocball->{nodetypes}) {
         $usergroup = $hDocball->{usrgroup};
         $list_type = $hDocball->{listtype};
         $other_types = grep { $_ ne $list_type }
         keys %{$hDocball->{nodetypes}};
   
         # there is no point in continuing further:
         # code currently supports only one section even
         # though the section is being tested within a for loop
         last;
      }
   [download]

CRUD and rebless operations

CRUD operations are triggered by sending a request for the appropriate URL.

• Create operations: Create operations are available using URL's with the following parameters:
  • op: set to "new". Most values for "op" have a corresponding opcode node. "new" does not. It is a built-in operator - see [Everything/HTML.pm|Everything::HTML::execOpCode]
  • type: set to the nodetype of the node to be created, i.e. 'sitedoclet' or 'pmdvdoclet.
  • displaytype: set to "edit".
  • addtogroup: set to the master document list for the node. For example, Master FaqList faqlist for faqlist, Master SiteFaqlet faqlist for sitefaqlet, and Master SiteDoclet faqlist for sitedoclet. For pmdev we would have master pmdvlist pmdvlist for pmdvlist, and so on. faqstring doesn't seem to be added to a master list even though we generate master doc string lists. Is this by intent or is it an oversite?
• Read operations: Read operations are available using URL's with the parameter setting "displaytype=display" and "node_id=NNNN" where NNNN is a valid node id. See triggers an htmlpage node. See the Nodetypes above for needed changes.
• Update operations: Update operations are available using URL's with the parameter setting "displaytype=display" and "node_id=NNNN" where NNNN is a valid node id. See triggers an htmlpage node. See the Nodetypes above for needed changes.
• Delete operations: In theory delete operations are also permitted but there doesn't seem to be any maint script for them (e.g. to purge doc lists or alternatively to prevent deletes when at least one doc is in a doclist). Nor do either SiteDocClan nodelet and SiteDocClan Procedures have delete (nuke) links. Did I miss something?
• Rebless operations: Rebless operations for site docs are triggered using a URL with the following parameter parameters:
  • op: "sdc_rebless".
  • node_id: the id of the node to be reblessed
  • sdc_rebless_to: the name of the new nodetype, e.g. "sitefaqlet" or "pmdvfaqlet"

The following changes need to be made so that CRUD operations will behave properly for the pmdev document collection:

1. sitefaqlet mainetance create (maint):

      #replace line 5
      #remove hardcoded value for author_user
      $$N{author_user} = getNodeById($node_id->{type}{writers_user})->{ti
   +tle};
   [download]

2. rebless (opcode): needs to be added (this code has been factored out of sdc_rebless)

      my $node     = $q->param('node_id');
      my $sNewType = $q->param('rebless_to');
      my $user     = $USER;
   
      $node =|| $NODE;
      my $hType = $node->{type};
      my $sOldType = $hType->{title};
      if ($sOldType ne $sNewType) {
         my $group
            = getNodeById($hType->{updaters_user})->{title};
         if ($DB->isApproved($user, $group)) {
            my $aReblessable = htmlcode('get_docball_data', ''
                , $group, 'usergroup')->{reblessable};
            my ($bCanReblessFrom, $bCanReblessTo);
            foreach (@$aReblessable) {
              if ($_ eq $sOldType) {
                 $bCanReblessFrom = 1;
              } elsif ($_ eq $sNewType) {
                 $bCanReblessTo = 1;
              }
            }
            if ($bCanReblessFrom && $bCanReblessTo) {
               Everything::printLog('Rebless '. $user->{title}
                 . ' #'.  $node->{node_id}
                 . ' - '. $node->{title}
                 . " from $sOldType -> $sNewType");
               $hType = getType($sNewType);
               $node->{type_nodetype}=$type->{node_id};
               updateNode($node, $user);
            }
         }
      }
   [download]

Seeing new nodes and recent changes

Both Recently Active Threads and Newest Nodes use Newest Nodes Settings to decide which nodes to include. To make changes to the pmdv docball visible to developers we need to add a setting key for each pmdv docball nodetype and update the settings storing nodes that are visible:

1. nodetypes, nodetypes_pmdev, nodetypes_cabal: append "pmdvstring,pmdvlist,pmdvdoclet,pmdvfaqlet"
2. pmdvstring: PmDev Strings,Master pmdvstring pmdvlist,1
3. pmdvlist: PmDev Lists, Master pmdvlist pmdvlist,1
4. pmdvdoclet: PmDev Doclets, Master pmdvdoclet pmdvlist,1
5. pmdvfaqlet:PmDev Faqlets, Master, pmdvfaqlet pmdvlist,1

To see recent updates (rather than new nodes), one needs to look at the Recent Janitorial Edits page. This page needs to be modified as follows:

   # replace lines 19-21
   my $hAllDocballs
      = htmlcode('get_docball_data','',undef,'all');

   # replace lines 24-27
   my $bFiltered = $q->param('Wi') && $type eq 'wiki';
   if (!$bFiltered) {
      my $type
         = getNodeById($edit->{edithistory_id} )->{type}{title};
      my $sDocball;
      foreach my $k (keys %$hAllDocballs) {

         # if $type is one of the nodetypes for this docball
         # then test the RJE filter param for this nodeball

         if ($hAllNodeballs->{$k}->{nodetypes}{$type}) {
            my $sParam = $hAllNodeballs->{$k}{RJEfilter};
            $sParam = $k unless defiend($sParam);
            if ($q->param($sParam)) {
               $bFiltered = 1;
               last;
            }
         }
      }
   }
   if ($bFiltered) {
[download]

Edit history

Edit history is created when docball nodetypes are saved via doclist edit page or sitefaqlet edit page, both of which call handle_node_edits to save changes. History is displayed either via the Recent Janitorial Edits or by a URL where "displaytype=edithistory". This triggers the standard edit history page for nodes: node edithistory page.

The necessary changes to support the use of these pages for other docballs has already been discussed above in the following sections:

• Editing and display of nodetypes
• Seeing new nodes and recent changes

Annotating other documents

The site documentation system can use the htmlcode node showsitedoclet to embed a site document into another page. This feature is sometimes used to provide humorous commentary about user groups and access rules.

At some pont we may decide that it would be helpful to annotate design elements like dbtables, nodetypes, and htmlpages with some commentary about their role in the larger system. However, first we need to have some documentation suitable for that purpose.

Our present primary concern is making it easy to create and keep track of documentation. This feature applies more to how documentation is used so it isn't essential to short term goals. Thus there are no immediate plans to modify showsitedoclet to work with anything but sitedoclets.

Pmdev documentation user interface

The SiteDocClan provides a number of different ways for users to work with documentation:

• SiteDocClan Procedures - provides links that can be used to create documents.
• SiteDocClan nodelet - provides links for reblessing nodes and a link to the Master FaqList faqlist
• Node lister can be used to view a list of all the nodes in a nodegroup. One can click on any node in the list to view it.
• The upper right corner of a node display page contains a link to edit the node.

The organization of the pmdv docball user interface will of course evolve over time. Initially, we can just place the following links at the bottom of the PmDev Nodelet using the following code changes. (Note: changes have been designed with reuse in mind. Hardcoded data that should be moved out to settings or database fields has been consolidated and marked.

1. PmDev Nodelet:

       #replace line 73: ); with
       htmlcode('genDocballEditorLinks,'', $NODE, $USER,
         'pmdev'));
   [download]

2. genDocballEditorLinks (htmlcode): add node

      my ($node, $user, $group) = @_;
      return '' unless $DB->isApproved($user,$group);
   
      my ($history,$rebless,$masterlist,$create) = ''x3;
      my $hType = $node->{type};
      my $sType = $hType->{title};
      my $hDocball = htmlcode('get_docball_data',''
         , $group, 'usergroup');
   
      #generate history links
      $history= linkNode(getId($node), "Node history"
                         , {displaytype => 'edithistory'}
                         );
   
      #generate reblessing links
      my $aReblessable = $hDocball->{reblessable};
      if (grep { $_ eq $sType } @$aReblessable) {
          my @links;
          foreach my $key (sort @$aReblessable) {
             next if $key eq $hType->{title};
             my $label = $hNodetypes->{$key}[0];
             my $link  = linkNode($node, $label
                { op=>'rebless', rebless_to => $key }); 
             push @links, $link;
          }
          $rebless= join " ","Rebless as ",@links;
       }
   
       #getnerate master document list link
       $history= linkNode(getId($hDocball->{masterlist})
                          , "Master Lists"
                          , {displaytype => 'display'}
                         );
   
       #generate create links
       {
          my @links;
          foreach my $key (keys %$hNodetypes) {
             my $aData = $hNodetypes->{$key};
             my $label = $aData->[0];
             my $hLinkParams = { op=>'new', type => $key };
             my $iGroupNode = $aData->[2];
             if ($iGroupNode) {
                $hLinkParams->{addToGroup} = $iGroupNode;
             }
             my $link = linkNode($node, $label, $hLinkParams);
             push @links, $link;
          }
          $create= join " ","Create: ",@links;      
       }
   
       return join ' | ', grep length($_),
           $history, $rebless, $masterlist, $create);
   [download]

Future steps

Here are some things we may want to consider after we implemented the above:

• refactor code so that all data affecting the features of a docball that isn't yet centralized is centralized in one place.
• improve display of diffs and edit history - can we make it as good as vcview's blame display mode? mediawiki's version history mode? Or do we have that feature already and it isn't documented?
• reevaluate non-use of showsitedoclet for pmdv. Do we want to use pmdev documentation to annotate core design elements like dbtable or nodetype pages?
• move hardcoded data out of htmlcode into dbfields or settings

Security issues

As I was reading through the code I noticed a few places where the code bypasses the permissions defined for each nodetype in the database:

• handle_node_edits is bypassing the node base's security checks and calling access rules directly rather than passing them through Everything/NodeBase.pm. This architecture has forced the creation of several access rules that hard code group membership security rules already stored in the database and could in principle violate them: (CanSdcEdit,CanPedagoguesEdit, CanQAEdit.
• handle_nn_settings uses values stored in settings instead of the database to decide if a user may view a set of node types. Of particular concern is the nodetype_all and canedit_nodetypes settings stored in Newest Nodes Setting. nodetype_all adds all of its nodetypes to the legal viewing list without checking to see if that nodetype is readable by the user.
• Newest Nodes Setting has a setting canedit_nodetypes is used by the isJanitorial accessrule and editor_vote htmlcode node to accept or reject an edit. Neither consumer node appears to supplement this check with a check on the user's group membership via $DB->isApproved.