(Note: this node is copied over from a response to a question (Perl apps and version control) on applications and version control that turned into an essay on development and deployment practices. I've copied it here and expanded it somewhat.)

These are some general recommendations here from my experiences (both bad and good) in managing builds and deployments; specifically, in making the whole process from initial development to live deployment as automatic, simple, and foolproof as possible.

Version control

From the standpoint of production and change control, a version control system is a must. Not just because it makes it easy for the programmers, but because it makes it easy for the change manager to track what is and isn't ready to be pushed to production, and provides operations with something to roll back changes if a new push is bad. With version control you have the option of being able to precisely undo or redo any change you've made - this can be really handy if (say) you've added a new feature, which works fine, but messed up a configuration setting. With version control, you can back out the bad stuff and keep the good (most of the time anyway).

In addition, you can use the branching and tagging features of modern revision control systems to maintain both a development and production version of the code, and precisely mark points in the development timeline for change review or deployment. Branches are particularly useful when you have code that changes relatively frequently, but which is deployed much less often. A branch can allow you to have a version of the source which is running in production that can be patched to include high-priority fixes without requiring everything currently under development to be included as well. Tags can also be used to mark particular points in the development timeline (for example, all the fixes up to a particular point can be tagged as the "1.0" version; the 1.0 sources can be checked out via that name, rather than by a specific date and time.

As far as choosing a version control system goes, there are a large number of possibilities (see List_of_revision_control_software for a fairly comprehensive list) You'll most likely see open-source alternatives (commonly Subversion, git, and Mercurial), though some prefer proprietary ones (ClearCase and BitKeeper). Your choice may be dictated by who you're working for; for instance, (at least as of 2009) Yahoo! uses Subversion; the company I currently work for uses Mercurial; the Linux kernel project uses git. Many of the modern revision control systems understand how to interoperate - as an example, there's a very good git-Subversion bridge that lets you do your work in git, but commit your changes to a parent Subversion repository. (Do not try to fight the "but this revision control system is better" battle: interoperate if you can, but make sure your commits end up where they ought. If you are running your own private repository and lose it because of your development machine having a disk failure, you'll only have yourself to blame.) If you have your own choice, either git or Mercurial will probably end up working better for you, but anything is better than nothing!

If you are doing this solo, you must must must make sure you have good backups of your revision control system. If you're working for someone else, make sure that they are backing up your revision control files.

Tagging and branching techniques

There are hundreds of ways to set up branching and tagging for projects; the guiding principle for your choice should be "what is simple enough that I will always do it, and complex enough to adequately handle our needs?" The following is a set of things to think about and some sample ways of addressing those things.

Branching

You should consider setting things up to be able to create at least one branch for production and one for development. Most modern development systems do what are called merges to bring branches into sync. Different systems handle branching differently. Mercurial can create anonymous branches containing a specific set of changes; git is practically all branches all the time- it's very easy to create, use and discard branches in git. Generally, you're going to want to make sure that your branches (especially the production branch) stay carefully separated until you merge at carefully-defined points. One (simple) way to do this is as follows:
  1. Create the repository, and do enough development to get to your first deployable milestone on the main branch.
  2. Tag the main branch at this point as your first release (whether this is a codename or a "1.0" or the like is up to you), and create a new staging branch (we'll talk about staging in a minute) with the same name.
  3. Deploy the staging branch to your staging test machine(s) and see how it goes. In the meantime, continue development on the main branch. If you find bugs that need to be fixed on the staging branch, fix them there and merge the fixes one by one back into development as you go. If you discover you've made a big error that will require a lot of fixing, you can do one of several things:
    1. Stop development until the staging branch is fixed, and merge the staging branch back to the development one when the repairs are completed. This will usually be the simplest way of handling it, and least likely to cause problems when merging back to the main branch (merge errors). This is probably best if you're resource constrained (that is, you need to deploy soon, and you have very few people working on the code, or commits seldom happen).
    2. Abandon the staging branch (in git, it's possible to actually delete the branch as well) and re-branch after the fixes are done. This has the advantage of not needing to make a big merge back into development, but may lay you open to not-ready-for-prime-time changes in development getting pushed into the staging branch. Good test coverage is critical for this to work, and even in that case it's risky.
    3. Allow development to continue on the main branch while the stging branch is fixed; create a new development branch by merging the (working) staging branch and the development branch into a new working branch when the changes are complete on staging. This puts a bigger load on the developers to ensure that the changes that have merged in from staging are compatible with what they're doing.
    4. Branch the staging branch, fix the problem, test the fix in the new branch, and then merge the staging fix branch back to staging and development both, optionally removing the staging fix branch afterward. This leaves the staging branch stable so any other problems can be found and fixed while the big staging fix gets done, and allows development to continue as well. Since you have the staging fix isolated in a single branch, development changes can now be merged into the staging fix branch; if there are problems, you know they are related to specific development changes and can fix those.
  4. At this point you have a stable staging branch with any outstanding just-for-this-branch problems fixed, and those fixes merged back into a development branch. Tag the staging branch, and branch it to a production branch. This branch should for all practical purposes be treated as immutable. If there are production problems with the code on this branch, you will not patch it here, but in the staging branch, then create a brand-new production branch. Your production branch is only for deployment!
  5. Check out this production branch and build it/deploy it on your production system - we'll talk about that process in a bit. You should at this point have something that will work on production without problems. If you do have problems, execute the backout plan, and return to testing on the staging environment to figure out why you didn't see this problem, set things up so it occurs, and then fix it. Merge the fix back to development, and create a new production branch to deploy from again.
This may seem complicated, and you may find a simpler process will work for you, especially if you're a solo developer. You may be able to get along with a single branch, tagging to denote each staging release and candidate production release; it will be simpler, however, to use branches to get better isolation of the different parts of the development and deployment cycle. The key here is coming up with a consistent approach that you will always follow.

Staging

"But it worked on my machine..." This is the last thing you want to hear (or worse, say). Development and production will always be different environments, and you need to make sure that when you are ready to move something to production, you've tested it on something that's as similar to the production system as possible. This allows you to pretend you pushed the code to production and then try it out. If you have a good relationship with your users, you may be able to get them to try the staging system to ensure that the changes you've made don't break anything as far as they are concerned (and if it has, talk it over with them and find out how to duplicate the problem; this gives you a new test case).

Staging doesn't always require an exact duplicate of the production environment; often you can manage this with a virtual machine and a partial set of data if you don't have enough servers to let you try things live. Check your environment, and create a test data (a small sample database for example) to provide all the resources you need to run. I've had pretty good success with VirtualBox - it's easier to get running than Xen or VMWare, at least for me - I've gotten it running on Linux and OS X with very little trouble.

Testing

Third, you need a test suite. Unit tests and integration tests which are run when the build is done, and "live" tests that hit the server and verify that it's doing what it should be doing. Selenium is a very useful tool for testing web interfaces. If you have a REST server, you may be able to use WWW::Mechanize to test the backend. If you also have a custom client, Mojolicious::Lite is a grand way to put together a mock server that can send the data you expect, as well as fail predictably so you can test your client's error handling.

If you have all these, then you can consider using a continuous integration server (I like Hudson for its flexibility and easy of deployment) to monitor your version control repository and automatically check out, build, test, and deploy to staging whenever a checkin is done (or on a fixed schedule, depending on how quickly the codebase is changing - you of course have the flexibility to only deploy once a day, etc.).

If your test suite runs quickly, you can do near real-time turnaround on testing the software as it's developed. Two minutes is ideal; five minutes is the outer limit. If your test suite is longer than that, I recommend a couple things:

Hudson provides many, many plugins to customize the build process; I recommend using one of the "extreme feedback" ones (xfpanel or radiator) on a monitor hung up where everyone can see it. This way if the build breaks, there's a big display that shows that it has happened. Hudson can also Twitter, talk to IRC or Jabber servers, and even just send email when a build fails. Use whatever you need to make sure that all the developers know when the build breaks - and make sure they know that a broken build is Bad and needs to be fixed right away!

Deployment

The key to good deployment is to have a simple (hopefully scriptable) deployment process. Every manual step in a deployment adds to the possibility of a failure or error. Staged deployments will, most of the time, allow you a simple way to check your deployment before you push it live.

Hudson's capability to create and track deployment "products" can be extremely useful. If you deploy by creating a tarball (for example, you build a CPAN-like tar file to be installed), Hudson can archive the tarballs for you, and track their signatures so that a tarball can be matched exactly with the Hudson build that produced it. This is so useful that you should at least consider jiggering your deployment process to be tarball-based.

If on the other hand you deploy from a known-good checkout using a custom script, Hudson also has another plugin that lets you manually deploy from a given successful build (Hudson saves all the output from every build unless you tell it not to; you can specifically mark builds as "never throw away" to allow Hudson to "roll off" builds that you don't need to keep.

Deployments need to involve as little decision-making as possible to perform; they may require decision-making after they are complete to make sure everything's working right (though a good "live test" suite, if you can manage to create one, is a great backstop again finding out two days later that things are broken!). The fewer decisions that need to be made, the less likely they are to be made wrong. The release engineer (even if that's you) should be able to follow a fixed set of steps and be able to check that each step worked properly.

If you have databases, you need to create what Ruby on Rails calls migrations: small programs that transform the current state of the database into whatever new state is needed before the new release can run , and back again. If the migration to the new format could destroy data, it's imperative to take a backup; your reverse migration then becomes a restore.

Deployments, whether they are clever scripts that do most of work, or lists of things to do, must be tested before they are used in production. If you've set up your staging system to resemble production as closely as possible, then you should be able to non-destructively test your deployment procedures on staging - and it's a good idea to have your deployment engineer (if it's not you) use the procedures to deploy to staging; this lets you make sure that the procedures and migrations work in an environment that won't matter if you mess it up.

Final thoughts

If you can manage it, try to create a design which integrates directly into the OS on the deployment machine as little as possible. This decouples OS upgrades from your upgrades.

Use local::lib, or build your own Perl if necessary, to ensure that OS updates don't end up breaking your code. In addition, if you avoid OS integration, you may be able to use a "twinned" deployment: two copies of the software, one the "live" old version and the other a not-running new version. You can then shut down the old system and bring up the new (barring database migrations, for instance) without problems, "flipping" back to the old one if there's a problem.

Anything that could destroy data or transform it in a non-recoverable manner requires extra steps to back up the pristine data and restore it if necessary. You must test this prior to the real deployment to make sure you're not crossing a line of no return. There's nothing worse than migrating to a new setup only to find you have a really bad bug that you didn't find in testing, with no way back to the old system!

EDIT: fixed unclosed <i> tag, fixed bad Wikipedia link.

In reply to Development and deployment - some basic guidance by pemungkah

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.