Refactoring webcode to use templates

Bod has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Refactoring webcode to use templates by kcott (Archbishop) on Jan 31, 2021 at 13:02 UTC
G'day Bod, [Note:* Your question is very generic, which is fine. Accordingly, my suggestions are also generic: you'll need to adapt these to your requirements.]* "... written entirely in Perl ... HTML, CSS, JavaScript, etc hard coded along with the functional logic." Before doing anything else, you should separate all of these concerns. There should be no "HTML, CSS, JavaScript, etc" in your Perl code. You probably want: (templated) HTML in `.tt`; CSS in `.css`; JavaScript in `.js`; and so on. As with any sort of strict* rule, there will be exceptions; for instance, you may need to toggle a CSS `display` property in JavaScript to handle a "show/hide" function. Here's a few general pointers: Start a new project. Copy parts of existing code where necessary but don't try to modify the current application on the fly. Make decisions regarding software versions at the outset. Consider what's available to all end-users. Don't try to retrofit partial changes into existing code. Hard-code as little as possible: templates can reference other templates. Avoid anything that may require subsequent changes to core code: prefer config changes. Give careful consideration to specific and generic elements; e.g. `id` and `class` HTML attributes which may be referenced by CSS and/or JavaScript. Test extensively. This pays obvious dividends up front. It's also the gift that keeps giving (as you'll find out when an apparently innocuous change to one thing completely breaks something else that appeared to be quite unrelated). Be patient. Avoid the urge to deploy early. Eagerness to show off your new creation is perfectly understandable; just make sure you're ready and avoid egg-on-face syndrome. Your template would look something like this: `<html> <head> <title>[% title %]</title> <link ... [% base_url %]/css/[% css %] ... <script ... [% base_url %]/js/[% js %] ... </head> <body> [% header %] [% page_specific %] [% footer %] </body> </html>` [download] "What are the hidden pitfalls that could trip me up and suck up vast amounts of time?" Following my pointers above should mostly avoid these; I wrote them with this question in the back of my mind. In my (not inconsiderable) experience, one of the biggest pitfalls in any software project is: "I'll just leave that for now and fix it later."; which is often followed by the famous last words: "That should be quick and easy to do.". If it is indeed quick and easy, do it now; however, that often turns out not to be the case and shortcomings, or even bugs, plague many subsequent production versions. Get it right once, then spend your subsequent time on things vastly more interesting than tracking down bugs or trying to refactor your refactorings to get what you wanted in the first place. In closing, this sounds like an interesting project and I wish you good luck with it. — Ken	[reply] [d/l] [select]
Re^2: Refactoring webcode to use templates by Bod (Parson) on Feb 01, 2021 at 23:20 UTC
Start a new project Thanks Ken, That is probably the key I was after without knowing it when I asked the question... My guess is that starting a new project and taking the great advice already given, I will have a far better idea of the real problems I am facing. A couple of days ago, the idea of refactoring a working website looked like such a mammoth task that I had no idea where to start. Now, I at least have a starting point.	[reply]
Re: Refactoring webcode to use templates by 1nickt (Canon) on Jan 31, 2021 at 00:21 UTC
Hi, Some random thoughts for a vague question: Use a base template for your HEAD section (and footer?) and the start of the BODY down to a super-parent DIV. Include other templates therein when you render a given page. (If you use Dancer2 for your app framework this is set up through config and works automatically.) For a given page, start refactoring by creating the rendered page as a flat HTML file. Save this as the template, pare down to what needs to go in the main template above. Identify variable data and start inserting tokens. Switch the page over to TT rendering. Hope this helps! The way forward always starts with a minimal test.	[reply]
Re: Refactoring webcode to use templates by Radiola (Monk) on Jan 31, 2021 at 19:11 UTC
I very much agree with kcott above. Perl has several good web frameworks; perhaps the two most prominent right now are Mojolicious and Dancer2. (I'm partial to Dancer2 with Template Toolkit, myself.) Either will have the tools to help you separate your code from your design. My advice: Don't rush. Spend a little time, even a few weeks, reading about them and looking at examples. Then start a clean, new project and start reconstructing your site the new way. You can create an unmaintainable mess even with good tools: fragile dependencies between objects, spaghetti callbacks to and from your templates. The time spent researching tools and getting familiar with them will pay. Same thing with testing. Technology doesn't directly have much to do with how effective your site is. Dancer or Mojolicious won't directly lead to one more conversion (nor will most technology, despite what salespeople always say). People do that: you, as you gain more experience in your testing, or specialists that you might hire. Better technology makes it easier for you to do your jobs.	[reply]
Re^2: Refactoring webcode to use templates by Bod (Parson) on Feb 01, 2021 at 23:12 UTC
I very much agree with kcott above. Perl has several good web frameworks... I will be using Template because that is what is installed on the webserver and because that is what I am getting familiar with building another website. I started using templating very recently thanks to the good information and influence that flows from the Monastery. The website which I plan to refactor didn't exist 12 months ago as got put together in response to the Pesky Pandemic. Technology doesn't directly have much to do with how effective your site is Don't worry! I am well aware of that... We have set about improving CRO which will involve incremental site changes. I want to refactor to make the changes easier rather than to increase the CRO directly.	[reply]
Re: Refactoring webcode to use templates by bliako (Abbot) on Jan 31, 2021 at 00:57 UTC
In a previous comment on your quests (I will later find a reference) I suggested that template files must be fine-grained, performance allowing. I did not have A-B testing in mind but now that you mention A-B, this plan sounds appropriate for it. Without thinking too much about it, the idea is that a webpage consists of several areas. You want to test your site's CR by changing any of these areas's design and see the effect it has. A fine-grain-template website means that you have a lot of "knobs" you can change and see where CR goes. Replace that template with this, in combination with the other, etc. To make it a bit more concrete, let's say you have a site which advertises houses for rent. And you want to test whether showing picture of the house, instead of describing it in text, increases the CR. So get the house template and break it at the point where showing house pictures or text description. Eventually House (H) template which used to be solid now consists of H=Rest+Pictures-or-Text. The "fine-grain" templates means that P and T reside in different templates and that your web-serving controller which sends html to your client browsers picks any of them at random, builds a complete page and sends to the client browser. More formally, if your web page consists of areas A(let's say title),B(let's say house description),C(let's say price) and A has 3 templates, A1,A2,A3. And so does B and C. Then your controller must put a page together by choosing one of (A1,A2,A3) for A, and one of (B1,B2,B3) for B and similarly for C. Which can lead to combinations like A1,B3,C2 or A2,B2,C1 etc, all of these combinations form a complete html page which shows OK. Clarification: B1 let's say has pictures of houses but B2 has only text and B3 has both and the part 'B' is the template for showing a house. 'C' could be the template for showing prices, etc. So now you can create 3x3x3 combinations of webpages to serve to your client-browsers when they hit your site and collect back some feedback - ultimately the CR, but also time-of-stay, making-an-enquiry etc. Keep track of what you served to client-browsers and what their behaviour was. Analyse it and you can find some guidelines to enhancing the site. Caveats: The more breakpoints you have in a single webpage (i.e. the more "knobs" you have to play with) the bigger the Space for exploring. Don't split your page in 1000 parts each with 2 options, meaning 2^1000 combinations, to serve to 5 customers per year and expect to get sound statistical results! Analysis will be rubbish. Actually given the number of your customers a statistician can tell you the maximum number of different combinations you can have. To recap: identify in each webpage places which you can have optional layouts (show pictures or not, prices or not, even using font A or font B). Each of these layouts will be a sub-template. Picking one of each of the sub-templates an putting them together will give you a complete web-page to serve. Use your web-serving application (CGI, Mojolicious, etc.) to make the choosing and then to track the user with cookies. Quantify behaviour and when you have enough do some statistical testing of hypotheses : "Showing house pictures and hiding prices yielded most email enquiries" is this significant? Again, a CPAN module should be most useful here ... and of course a lesson to what our life amounts to: A-B rats. bw, bliako and clear those cookies often	[reply]
Re: Refactoring webcode to use templates by Polyglot (Chaplain) on Jan 31, 2021 at 08:42 UTC
For myself, I have found dividing my project into separate files really is helpful in organizing everything. In place of my real filenames, I've done a slight modification, but you can see the gist of it here: `require "${directory}MyCGI_HTML.pl"; require "${directory}MyCGI_HTML_Main.pl"; require "${directory}MyCGI_HTML_Preferences.pl"; require "${directory}MyCGI_MySQL.pl"; require "${directory}MyCGI_LANG.pl"; require "${directory}MyCGI_Other.pl"; require "${directory}MyCGI_DB-Export.pl"; require "${directory}MyCGI_FindReplace.pl"; require "${directory}MyCGI_SEARCH.pl"; require "${directory}MyCGI_AJAX.pl"; require "${directory}MyCGI_Valid.pl"; require "${directory}MyCGI_SpellCheck.pl";` [download] Naturally, the `$directory` variable holds the location of these files, which does not happen to be the /cgi-bin/ directory. It's a cinch to do a bit of versioning by simply copying the files to a new directory, updating them there, then changing this directory variable in the main CGI script. As you can see, the files each have their specific purpose. In the "MySQL" one, I put subroutines with all of the MySQL queries and operations that need to be done--at least, most of them. In the "ExportDB" one, several specialized routines for making database exports available are to be found, and it's a sufficiently unique subset of MySQL operations to merit its own file. These, of course, are only the Perl files. I also have, with similar nomenclature, separate .js and .css files. As I am working with AJAX, I put the Perl AJAX routines in a separate Perl file from the JavaScript AJAX functions, which would be in a .js file. It makes it very easy to find things, and works well for me. I don't use any framework like Dancer, etc. I do all of my own programming from scratch. I've found it's much less troublesome for me to troubleshoot my own code than to be dabbling with someone else's code. I don't even use jQuery--it's just too esoteric for me, and bloated as well. My own functions take a fraction of the bandwidth that jQuery-driven functions would. Two of the files, from my perspective, could be further subdivided to make them less unwieldy--the HTML and the MySQL. Both have a lot in them. Perl code tends to be compact compared to some of the non-Perl routines. At the top of each of the required Perl files, I might have something like this: `use utf8; use CGI qw(-utf8); binmode(STDOUT, ":utf8"); 1; sub exportdatabase { #INCOMING GLOBAL VARS: $statement, $db_export_file, # $cur_date, $cur_time, $OS, $table, $quest fork: { #...truncated.` [download] Notice that the file then just has the subroutines in it (below the `1;`), and attaches to the main script as if it were part of it, with all of the global variables shared, etc. If all of these subroutines were combined into the same file, it would work just the same. By dividing things, it just helps to be more organized. (I'm not even sure if I need the "use" clauses at the top...but have put them there for good measure, particularly when the file contains UTF-8 characters.) I embed variables within the main HTML template--which is itself put into a variable that is the return value of the subroutine. A sample might look like this: `my $template = <<TEMPLATE; <!DOCTYPE html> <HTML lang="utf8"> <head> <title>My Page Title</title> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf8"> $dragcss <!-- # S T A R T O F J A V A S C R I P T S H E R E --> $positionscript $AJAXJS <link rel="stylesheet" href="$cssfile"/> <link rel="stylesheet" href="$css_skin" title="Choice"/> </head> <body> <DIV class="fullpage" > ... Put the main content here. </DIV> </body> </html> TEMPLATE` [download] Hope this gives you a few ideas. Blessings, ~Polyglot~	[reply] [d/l] [select]
Re^2: Refactoring webcode to use templates by hippo (Archbishop) on Feb 01, 2021 at 15:31 UTC
`require "${directory}MyCGI_HTML.pl"; require "${directory}MyCGI_HTML_Main.pl"; require "${directory}MyCGI_HTML_Preferences.pl"; require "${directory}MyCGI_MySQL.pl"; require "${directory}MyCGI_LANG.pl"; require "${directory}MyCGI_Other.pl"; require "${directory}MyCGI_DB-Export.pl"; require "${directory}MyCGI_FindReplace.pl"; require "${directory}MyCGI_SEARCH.pl"; require "${directory}MyCGI_AJAX.pl"; require "${directory}MyCGI_Valid.pl"; require "${directory}MyCGI_SpellCheck.pl";` [download] Have you considered structuring your files as modules instead? 🦛	[reply] [d/l]
Re^3: Refactoring webcode to use templates by Bod (Parson) on Feb 01, 2021 at 23:15 UTC
Have you considered structuring your files as modules instead? This is exactly the sort of thing that I am moving away from...lots of `require` files. Thanks to the good influence of the Monastery, I am now using modules for new code. This whole refactoring project will revolve around rewriting `require` as `use` modules...	[reply] [d/l] [select]