amonroy has asked for the wisdom of the Perl Monks concerning the following question:

Dear monks,

I would like to hear your comments/critiques about my design of a perl program that will deploy updates for a database-driven web application. I can change everything excep the design of the web app itself.

About the web app.
It's the typical LAMP (Linux, Apache, MySQL, Perl) application. It's composed of CGI's running under mod_perl, lot's of perl modules and a database. All of the components change quite frequently: the software (sw) and the database (db). A change in the db typically involves a change in some pieces of sw. There are also sw fixes that are independent from sw-db changes. Some customers pay for sw-db updates others pay just for sw-fixes.

About the update mechanism.
There will be about 1,000 clients using this mechasim on a monthly basis. Each client has it's own installation of the web app in a dedicated server. They have to apply the monthly updates sequentially, meaning that they shouldn't apply the October update if the September update hasn't been applied.

I am thinking in implementing it in a way that it could be executed in the command line (similar to how you execute yum or apt-get), as a cron process or even with a web interface (in the future).

About the montly releases.
Each release is composed of three tar files:

  1. Changes in the sw related to db changes (sw-db.tar),
  2. sw bug fixes not relaed to db (sw-fixes.tar), and
  3. changes in the db: updates, deletions, insertions (db.tar)

The content of the sw-db.tar and sw-fixes.tar is just a group of .pm and .pl files that will be deployed in certain directories.

The content of the db.tar is an xml file with a section for DB table. Each table section in the xml file has an <update>, a <delete> and an <insert> section.

The process.

  1. Check in a config file the latest update applied , i.e. the config file might say something like "the last update was the July release."
  2. Based on the latest applied update, download via http, all the necessary monthly-packages. For example if the last applied update was the July release and the latest released update is the October one, it will have to download the August, September and October releases.

    Depending on the IP address of the client, the update server might not provide db and sw-db packages.

    Some clients are not able to connect their servers to the internet so the mechanism should allow for other means of distribution of packages, ie. bringing the files in a CD or something like that.

  3. For each downloaded relase package (e.g. aug, sep, oct) do the following:
    1. Deploy sw-fixes.tar and sw-db.tar updates by just un-tar-ing them.
    2. Deploy db.tar by unpacking the xml file, parsing it with SAX and apply the db changes. It's very important to avoid overrding any customization made on some fields of the db.
    3. Update config file stating that the monthly release has been applied.

The program should be able to stop at any give step and continue.

Thanks in advance,
-Andrés

  • Comment on Designing a monthly software and database update for a web app

Replies are listed 'Best First'.
Re: Designing a monthly software and database update for a web app
by Zaxo (Archbishop) on Oct 16, 2004 at 22:31 UTC

    You have some payment scheme for your packages, but your design does not incorporate any enforcement of that. You do mention IP address validation, but that is a very weak reed. How is that to work with the CD distribution you next mention?

    I don't understand why the intermediate xml steps are necessary. Why not write the db updates as a MySQL batch file?

    You should consider the system patch and make utilities, as well as version numbering instead of date-based upgrades. There are well-known perl modules which can help with each stage of your design.

    After Compline,
    Zaxo

      You're right, the payment scheme doesn't fit well with the CD idea. I am still not clear on how to go about that. I was thinking in giving up the idea of the CDs and just ask people to somehow download the updates by hand if their server is not hooked to the internet. They will have to connect from certain IP though.

      The reason why it cannot be a MySQL batch file is because the update algorithm is not very simple, it requires a lot of perl code to analyze when a field is considered customized and when it is not.

      The versioning number will be in this form: YYYYMMDD, thanks for your suggestion.

Re: Designing a monthly software and database update for a web app
by pg (Canon) on Oct 17, 2004 at 00:02 UTC
    "the last update was the July release."

    When you design, you design for the future. Your program should not care the frequency of the releases, but rather the sequence of the releases. Make the release named something like "July release" is not a good idea. You don't want to modify your code just because the frenquency changed.

    The other thing is where to get the files to tar? If you are not using any version control system, go get one; if you do use one, get whatever you want to tar from version control system, not certain directory or anything like this.