RFC: Application Configuration Management

I originally posted this on my blog, but realized this might be a better place to get the kind of feed back I am looking for.

I signed myself up to write application configuration management into the soon to be released $SUPER_SECRET_OSS_PROJECT. The goal is to have a single default configuration file located on the system (with conf.d/* style includes ) that would define each instance of a particular application.

An instance is simply a name and an easy way for the programmers and admins to talk about each install of a particular application. By simply knowing the instance's name the web application, whether it is a standard Perl CGI or uses FastCGI, mod_perl 1.x, or mod_perl 2.x, can essentially automatically configure itself based on the configuration method preferred by the admin. That's right the admin.

Often there exists a complete disconnect between the programmers and the system administrators responsible for actually making the app run. Programmers want a configuration setup and style that is easy to parse , while sysadmins want a system that is flexible and easy to use.

The configuration infrastructure that I'm building into $SUPER_SECRET_OSS_PROJECT should give everyone the best of several worlds. Here is the idea in a nutshell:

You have a central config file let's call it /etc/myconf.conf that will contain a entry for each instance of each application.
The instance information will contain either the entire configuration or enough configuration information to bootstrap the config engine enough to gather the rest of the config.
The application and any programs or associated cron jobs only need to be given ( via environment variables, command line options, or eek hard coding ) the name of the instance they are a part of to configure themselves.

This is probably best explain with a short example, here is a mock up of the central config file /etc/myconf.conf:

<instance foo>
ConfigViaFlatFile Config::Tiny /etc/apps/foo/foo.cfg 
</instance>

<instance bar>
ConfigViaParamBuilder Apps::Foo::Params
</instance>
[download]

The first instance foo wants to configure itself using the Config::Tiny module and the config information is in /etc/apps/foo/food.cfg

The second instance bar is saying that it wants to configure itself using ModPerl::ParamBuilder and to use the Apps::Foo::Params module to do so. There will also be ConfigureViaSQL, ConfigureViaLDAP, etc.

So far this is all pretty boring. The real exciting piece is that these two instances could very well be the same code base. One where the admin wants to configure it via custom Apache directives in the httpd.conf and the other using a simple flat text file for configuration. It doesn't matter to the application. You might be asking yourself how that could be. Time for another example:

use SSOP::Conf;   # For Super Secret OSS Project 
my $conf = SSOP::Conf-&gt;retrieve( $instance );
my $template = $conf-&gt;template;
my $dbuser = $conf-&gt;dbuser;

# Or by using a hash reference 
my $conf_ref = SSOP::Conf-&gt;retrieve_hashref( $instance );
my $template = $$conf_ref{template};
my $dbuser = $$conf_ref{dbuser};
[download]

That is all the application needs to know about, the "instance" name, to configure itself via several methods. Right now the plan is support Config::General, Config::Tiny, PerlSetVars in mod_perl 1 and 2, and ModPerl::ParamBuilder in mod_perl 2.x.

The infrastructure is written so that it doesn't matter to the application how the configuration is gathered and parsed, just so long as it has it. This frees the admin to use whatever configuration method makes the most sense for not only each particular application, but each particular instance of that application.

I think the most exciting aspect is that, provided you use a separately included httpd.conf for each of your instances, your behind the scenes programs, scripts, and cron jobs can share their configuration information. If you like PerlSetVars or ModPerl::ParamBuilder and want all of your app's config to live in your httpd.conf, you don't have to have a separate configuration scheme for your cron jobs.

You're probably saying to yourself, "Why the hell is he telling us this when we can't see the code yet." Well the whole purpose of this entry was to ask all of you which configuration file syntaxes they prefer to use, Apache style, .INI style, etc. so we know what to support out of the box. There are plans for a generic SQL backend and LDAP support. The system is built so you can expand on it yourself if there is something we don't support.

Comments, suggestions, rants, death threats, ideas, enhancements, etc. are most welcome. We want this system to be as useful to as many developers and admins as possible.

UPDATE:: Now that Gantry has been released there is more info here Configuration Flexibility

Frank Wiles <frank@revsys.com>
www.revsys.com

Comment on RFC: Application Configuration Management Select or Download Code

Replies are listed 'Best First'.
Re: RFC: Application Configuration Management by samtregar (Abbot) on Feb 12, 2006 at 20:25 UTC
Krang's configuration system supports instances much like you describe. It uses my Config::ApacheFormat module to manage them, which you might find useful. In general, I recommend you keep it simple. Unless you're absolutely sure you need something like ConfigViaWhatever I think you'd be better off skipping it. Choose a single relatively easy to learn config syntax and stick with it for all your instances. -sam	[reply]
Re: RFC: Application Configuration Management by brian_d_foy (Abbot) on Feb 12, 2006 at 20:54 UTC
What problem are you trying to solve? Who's the "admin"? Is that the system admin, the database admin, the application admin? If I'm managing an instance for my use, does that make me the admin? Centrally configuring everything for everyone has its drawbacks. It's better to provide defaults then let the users override and add to that with additional configuration. Of course, I have this oddd notion that system operators exist to serve users, not dominate them. If you make it so the users have to track down an operator to get changes they should be able to make themselves, users will look for something else to use. As for the configuration syntax, pick a single, simple one and stick with it. If you can configure everything with one syntax, another syntax doesn't buy you anything other than headaches, additional docs to confuse the user, and more ways to answer user questions. -- brian d foy <brian@stonehenge.com> Subscribe to The Perl Review	[reply]
Re^2: RFC: Application Configuration Management by ides (Deacon) on Feb 13, 2006 at 00:15 UTC
All of the points so far are very valid. But I guess I didn't make myself very clear. I'm not looking to support a position that you should use different config syntaxes just willy nilly. What I'm am essentially building is a config file interface, much like DBI that would all you to divorce how a particular application is configured from the actual code. This isn't for any one particular application, but a library for use in building applications. A config file abstraction layer if you will. Code cares about very few things when it comes to configuration information. Most in fact are simple scalars of information. Database name, password, number of items to show on a page, etc. Some are lists such as allowed IP ranges. But very little else. Looking at all of the various config file parsers on CPAN show that they all basically provide the same types of operations, just with a different syntax for each ( well most, some have exactly the same syntax, just a different API ). By admin, I mean whoever is installing, configuring, and/or maintaining the system once it is in production. This could be the same person who wrote it, a different person in the same department, or someone in another country who just happens to be using your software. I would hope that people would stay consistent within their own realms ( their home, their department, their company, etc. ). But I see this as actually helping with that. Say my shop likes to configure everything in a SQL database. We've written a neat config management tool to make and track changes, etc. Along comes some application we need to fold into our mix, but it uses a INI style flat file configuration. Is having this one app use a different style the end of the world? No. Is it a pain? Sure. Is it a reason to not use the application? Could be, depends on the exact scenario. Wouldn't it be nice if I could use basically whatever configuration method *I* choose and have the application not care? One situation I run into quite often is that you have a web application written in say mod_perl and you are using PerlSetVars to pass in configuration information. Being a web application written in mod_perl this seems perfectly resonable and expected. Later in life you realize you are going to need to run a couple of cron jobs to do some batch processing in your app that you had not originally forseen. It would be really nice if the cron jobs could simply share the same configuration information as the web portion, but that would non-trivial. So you end up either rewriting the web app's config code to use a flat file format they can both share or duplicate your config via commandline arguments to the cron jobs. In a situation where you have only a few applications to maintain this isn't hard to remember what is where and why. But now expand that situation to where you maintain 75+ different applications written by several different people over the course of the last 7 years. Now it's pretty darn easy to change say a database password or other piece of configuration information in your Apache config, but neglect to make the change in your crontab. That one cron not running could be difference between having a job and not having one. ;) Here is another example that I hope helps illustrate my point. Suppose you have an application that originally was only supposed to support one instance. Not that there are any real limitations in the application, just that it was never dreamed that you'd want to run multiple of them. So following with that the programmer used a simple configuration system you don't like. For the purposes of this example, let's say we don't like XML config files, or Apache config files, or GUI config apps with a binary registry, whatever... it doesn't matter. For whatever reason you either simply don't like the config system, there are problems with it when using multiple instances of the application, or it simply does not fit in with the overall scheme you are using in a particular realm. Now suppose you need to run hundreds of them. Maybe in a web hosting scenario. You've got this really great customer management application that handles nearly everything for you in a database, but now you have to write special code to build these text config files, publish them onto the server, etc, etc, etc. If the application had been written to where it was "config system agnostic" you would simply need to change the instance entry in that one file and presto your preferred way of configuration is available. The goal is to support as many different configuration "systems" as possible. It probably will not be possible to support every feature of every variant of everything out there. But ya' gotta start somewhere. :) Frank Wiles <frank@revsys.com> www.revsys.com	[reply]
Re^3: RFC: Application Configuration Management by dragonchild (Archbishop) on Feb 13, 2006 at 03:50 UTC
The reason DBI works is that it doesn't do anything. It provides a API, and a very simple one, at that. Plus, it's working on top of an extremely well-defined (if poorly thought-out) language - SQL. If you don't believe me, think about what it would take to build a DBD for DBM::Deep. It seems to me that you are getting ahead of yourself. What you really want is a way of standard way of querying configuration, regardless of how it's stored. Define that and you can use DBI as your configuration manager with people providing DBD's for each configuration method. Good luck. My criteria for good software: Does it work? Can someone else come in, make a change, and be reasonably certain no bugs were introduced?	[reply]
Re^2: RFC: Application Configuration Management by ides (Deacon) on Apr 26, 2006 at 14:51 UTC
Hi brian, just an FYI, but now that everything is all released you can see what I was talking about in this post. Frank Wiles <frank@revsys.com> www.revsys.com	[reply]
Re: RFC: Application Configuration Management by exussum0 (Vicar) on Feb 13, 2006 at 13:27 UTC
Being a java head, take a look at Spring and its configuration format. It covers a lot of that object instanciation/configuration stuff, where you need access into the attributes of objects as well. There's an XML and .properties way of configuration now. There is a 3rd (or 4th) way that they are working on to pleaes the people, like me, who want a more natural configuration (tag) language using XML. springframwork.org. Don't get too lost in all of the options it has. It's java heavy, but there is a lot of cool concepts that could be ported to perl.	[reply]


Pathologically Eclectic Rubbish Lister
	PerlMonks