Config File Shuffle

pileofrogs has asked for the wisdom of the Perl Monks concerning the following question:

I'm working on an app and I want to include sophisticated config files processing. Here's a list of what I want:

all config vars can be set in any config file and the command line
the app can read arbitrary number of config files
values set in first config files read will be overridden by values in later config files and the command line will have the highest priority
the user can add and remove config files from either the command line or any config file. A config file could remove itself.
the user can specify a whole new order for the files to be processed in, again at the command line or from any config file

Dizzy yet?

The hard part of all this is the chicken-and-egg of letting the config files decide which config files to process.

For example A.conf,B.conf and C.conf are the normal config files in order. (A.conf overrides B.conf, B.conf overrides C.conf) C.conf adds D.conf, B.conf removes C.conf and adds E.conf. So, in the end, we should get the values from A.conf, B.conf and E.conf. I got confused just writing this example!

My best idea so far is to write a big fat algorithm that starts with the basic list of config files and...

reads the files in the list
alters the list of config files
checks to see if in the first read through it read something it shouldn't have
alter the list again
read the files in the list
repeat until the list is stable

And only then start reading config files for the purpose of gathering the other config values.

Does anyone have any bright ideas of how to handle this?

Thanks!
--Pileofrogs

Update: I never knew the right names for the XY Problem and YAGNI. Groovy!

I think, yeah, you're probably both right in that I'm suffering from both XY and YAGNI.

To speak to the XY issues, I've been working with CPAN::Mini::Inject, but the configuration files and command line handling could be better. Specifically, you have to put some variables in config files and others have to be specified on the command line. The command line is always really long. I thought it would be better if you could (1) specify any variable either on the command line or the config file and (2) if values set in the global config files (/etc/mcpani) could be overridden by values in more specific config files ($HOME/.mcpani/config). Some values tend to be the same no matter what you are doing, others tend to be different from user to user, but the same across that user's projects. Others tend to be different from project to project. I've emailed with the author, and he liked my ideas so I'm working on implementing it now.

Now, we get to the YAGNI. What if I decide I don't like what the sysadmin puts in /etc/mcpani? Or, maybe I realise that half my projects use 5 of the same variables, so maybe I want to include a file?

In the reality of the problem, I should probably just say "These are your config files, if you don't like the values, in the global config, override them yourself. If you have the same values come up over and over, deal with it."

However, in the context of monks chewing the fat, this problem seems interesting. I know certain apps do something like this, for example, vim reads /etc/vimrc and $HOME/.vimrc and a zillion other config files as well...

Comment on Config File Shuffle

Replies are listed 'Best First'.
Re: Config File Shuffle by roboticus (Chancellor) on Apr 24, 2006 at 17:55 UTC
Pileofrogs: Not to be difficult, but if it makes you dizzy (and my head spin), how are the users going to be able to effectively use it without shooting themselves in the foot? I'm all for config files, but I think you're about to make things hard for yourself. Perhaps you're having an XY problem? --roboticus	[reply]
Re: Config File Shuffle by philcrow (Priest) on Apr 24, 2006 at 18:36 UTC
Gantry::Conf does many of the things you want. But, it doesn't deal with command line overrides of any kind. Nor does it provide support for 'A config file could remove itself'. (I have to admit that I can't imagine a use for that.) Gantry::Conf does let you specify conf in a single place for lots of apps or in lots of places for a single app. It allows you to share conf across apps (or not, of course). It allows for precedence based overrides and many other things. It can even fetch conf via https (or http if you don't have to worry about security so much). That allows multiple servers to share conf. It also allows separate instances of the same app to share their common conf while keeping some of their conf instance specific. Further, it's extensible, so you can write backends to fish conf from your favorite data store (ours tends to be flat files in Config::General format). <disclaimer>Gantry::Conf was conceived by my boss and written by him and others in our shop, including me.</disclaimer> Phil	[reply]
Re^2: Config File Shuffle by pileofrogs (Priest) on Apr 24, 2006 at 18:52 UTC
Sweet! I'll check it out! For an example of when a file might want to remove itself, think of xinitrc, or yum and the /etc/yum.repos directory. Each has a batch of config files with a value for "use me" or "don't use me." Thanks! --Pileofrogs	[reply]
Re: Config File Shuffle by dragonchild (Archbishop) on Apr 24, 2006 at 18:00 UTC
An example of these config files would help a lot, as would the application you're writing this configuration process for. Along with an XY problem, you might be suffering from a lack of YAGNI. My criteria for good software: Does it work? Can someone else come in, make a change, and be reasonably certain no bugs were introduced?	[reply]
Re^2: Config File Shuffle by mantadin (Beadle) on Apr 24, 2006 at 18:33 UTC
Supposed, we got the YAGNI case, and all you need is all config vars can be set in the config file and the command line the command line will have the highest priority the user can set the config file location at the command line In that case, you could do it like this: `use Getopt::Std; my %opts; getopts( 'dhc:', \%opts ); # usw, -c for cfg-file $opts{'c'} and $opts{'c'} !~ m{^/([\w\.]+/)[\w\.]+$} and &usage; my $cfg = $opts{'c'} ? $opts{'c'} : '/default/path/to.conf'; open CFG, '<'.$cfg or die; while(<CFG>) { # do sth like ... /^\s(.?)\s: (.*)$/; $cfg{ $1 } = $2; } foreach (keys %cfg) { $opts{$_} and $cfg{$_} = $opts{$_}; }` [download] You can do this "overriding" as many times you like, using a list of config file locations. The script would eat one file after another, starting with the "weakest", most global one, and finishing with the command line options. The overriding process could be done in the manner indicated in the code above.	[reply] [d/l]
Re: Config File Shuffle by hesco (Deacon) on Nov 09, 2008 at 00:52 UTC
Yeah, your post is over two years old, but through the magic of SuperSearch I just stumbled across it. Some of what you describe above I wrote as Config::Simple::Inherit, which then got subsumed in Config::Simple::Entended (which I still maintain, I guess, until the author of Config::Simple wants to incorporate the new methods I use regularly in his module). Anyway, it handles the inheritance of configurations for you. And Extended also offers a method which will parse an incoming url to determine which configuration you want to use so a single installation of the code base can host multiple instances of an application simply by reading different configuration files. Long range I will eventually also add support for heredoc to ::Extended, as well, so it can be used to feed copy into a webform. Check it out if you think this might be useful. Its been uploaded to CPAN. -- Hugh if( $lal && $lol ) { $life++; }	[reply]

Sweet!