http://qs1969.pair.com?node_id=506585


in reply to What is YOUR Development Process?

I do web apps. My ideal world:
  1. Everything is under source control I use Subversion, and recommend it highly.
  2. Everything has an automated test that, at the bare minimum, verifies that it will compile. Preferably, everything is tested to at least 95% coverage, with 99% being the goal.
  3. Everything is developed and tested in the developer's environment. This may be a separate machine or it may be a separate port on the same machine. I've done both and there's no benefit vis-a-vis development process. If you check something in, you're asserting that you have done at least some manner of testing. In other words, HEAD will, at the very least, compile.
  4. Either the full test suite is run whenever a checkin occurs (ideal) and/or it runs nightly on some smoke server (good). Best is both.
  5. Did I mention that EVERYTHING is under source control? This includes anything that your application might need in order to run.
  6. Any external items (CPAN modules, system commands, etc) and their versions (min and max) are marked as dependencies somewhere in the installation script. It is so annoying when you try and install something and it breaks because it forgot a dependency. (The automake19 port on FreeBSD 5.3 has this problem.)
  7. Everything that happens in the application is logged. Every request, who made the request and from where, how long it took, and if there were any errors. If you have the space (and you probably do), log the response as well. Tracing errors is a lot easier when you can see exactly what your app sent back in response to what.
  8. If an error occurs on prod, it should, at the very least, email someone. Preferably, someone gets paged. Customers love it when you call them 15 minutes after the site says "I can't let you do that, Dave.", apologize for the inconvenience, and inform them of both a workaround and that the problem has been logged as a bug. If you have a pager, rotate it. That way, everyone gets experience.
  9. If you don't have code reviews, write each other's tests. You have to, at the very least, have working familiarity with the APIs that your colleagues are working on. Otherwise, how can you possibly suggest improvements at the time when it's cheapest to make those changes?
  10. The dev machine/dev port is for developer integration testing. It's also useful for demonstrating to management how the new unfinished feature will look like.
  11. There is a completely separate environment called "test". Ideally, this is on a separate machine in order to test the upgrade protocol, but it doesn't have to be. This is the place where the testers test and, most likely, UAT will occur here.
  12. Every modification to the application has a change request associated with it. This means features, enhancements, bugs, and retirements. They are all change requests and need to be identified. All checkins (ideally) are for a single change request and should be marked with the request ID.
  13. When doing an upgrade, every action is scripted. This means that if a table has to change, the ALTER TABLE statement(s) are in a script, preferably identified by the change request ID that it is handling. If the upgrade is complex, there should be a single master script that will call all the other scripts to do the work. These scripted actions are how you move changes from dev to test.
  14. There is a prod environment. This will probably be multiple machines. No-one is ever logged into prod. Ever.
  15. There is one single person responsible for any given install. Ideally, each person gets to handle a prod install. That person is the only person allowed to "pull the trigger" on the install. In other words, they are the only person logged into that machine during the install window. And, this is the only exception to the prior rule.
  16. There is a maintenance window on prod where the client understands installations may be occurring. Unless it is a crash-bug, this should be the only time changes are made to prod. Period.
  17. Let me repeat - Prod is sacrosanct. If you need to find something, look in the logs. That's what they're for. If you can't find it there, log it for next time. Every time you touch prod, YOU WILL BREAK IT.

My criteria for good software:
  1. Does it work?
  2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?

Replies are listed 'Best First'.
Re^2: What is YOUR Development Process?
by IOrdy (Friar) on Nov 08, 2005 at 03:11 UTC
    "There is a maintenance window on prod where the client understands installations may be occurring. Unless it is a crash-bug, this should be the only time changes are made to prod. Period."

    At my workplace each machine on the farm runs two web servers one with the production code and the other with the final QA code. Come release time once a week we switch the network so the final QA web servers become production then update the old production web servers with the next QA version in the pipeline.

    No maintenance downtime and the final QA code is checked on the exact same production machines.
      Do you also mount /usr/local so that both servers mount from the same physical location? That way, you guarantee that the perl installs are identical ...

      My criteria for good software:
      1. Does it work?
      2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?
        No idea! After peer review I just commit my code and submit a ticket to QA with a description of the change. Beyond that it's kinda blurry as Operations don't let you peek over the fence much and I'm no server side guy :)
Prod to test
by Anonymous Monk on Jul 09, 2006 at 02:30 UTC
    I'm just another anonymouse sysadmin, but I wanted to make one comment: There needs to be a way to back-port the prod environment into test (and possibly even to dev). This way you can recover from situations where the upstream environments have become gunked-up with oddities that just naturally happen during the dev-test-prod cycle. And this backport from prod to test needs to be either a script, or a full backup/restore cycle, to ensure complete fidelity to prod.