http://qs1969.pair.com?node_id=525194

cbrandtbuffalo has asked for the wisdom of the Perl Monks concerning the following question:

Esteemed Monks,

We support some fairly complex web applications and lately we're running into some issues with our config infrastructure. I got some help recently when I asked What config info belongs where for web apps?; thanks to those who responded.

Now I'm trying to define a better config system, but I can't find any best practices or guidelines beyond simple config file processing.

Here are my needs:

Here's a draft of a best practice (untested), with many questions:

### Shared config info -- "machine scope" ### # A local module that loads the shared config info. # Is there a way to pass in a project name in a 'use' # that is mod_perl safe? use My::LocalConfig qw(project_name); ### App-specific config info -- "machine scope" ### # Assuming 'config_path' is a function exported by the LocalConfig mod +ule. # Or should it just set a variable? # What sort of variable and where can it be set so it's ready when # the require runs? # Would a 'do' be better? # Should the project_name be passed in here as a parameter? require config_path . "/my_config.pl"; # Or should we use a Config module here? # Do various config modules allow a function call when you set # the path? # Are there Config modules that allow some logic in them? # We have several cases where we set # variables differently based on dev, QA, or prod environments. ### Session config info -- "session scope" ### # A module that just grabs relevant $ENV info and # provides it via a method or methods. use My::SessionConfig; # then my $session = My::SessionConfig->new(); my $current_userid = $session->current_user();

What are you doing for your config environment?

Do you have custom modules to handle all of this?

Once I nail down the best practice or a functioning system, I'd like to follow some of TheDamian's advice and hide a bunch of this to make it easier to use. But we've already been burned by some cleverness in our mod_perl environment, so I'll want to be careful.

Replies are listed 'Best First'.
Re: Configuration Best Practices for Web Apps
by perrin (Chancellor) on Jan 24, 2006 at 17:55 UTC
    A deceptively simple question.

    One piece of advice, before I suggest an approach: using a massively complicated config module that can do all of this is probably less useful than using a simple one and having some wrappers that cause the specific behavior you want on your system.

    I suggest an OO API more like this:

    use My::Config; my $config = My::Config->new(project => 'GuestBook'); my $db_password = $config->get('db_password');

    Why an OO API? Because it avoids issues with re-using globals in persistent environments. What if you want to use config variables for more than one app in a single script (or handle requests for multiple apps in one mod_perl process)? You can take care to reset your globals, etc., but it's a pain.

    Using an OO API means you have to deal with getting the config object frequently, but it should be pretty fast if you keep the parsed files in memory.

    In terms of how to set up your actual config files, I'd suggest either having multiple files (e.g. system.conf, app1.conf, app2.conf) and merging them as hashes in memory (so app1.conf overrides system.conf when project eq 'app1') or using a config module with built-in support for overrides. We currenctly use Config::ApacheFormat, which lets you do things like this:

    DBPassword overr1de_me <App Falcon> DBPassword chewb@cca </App> <App DeathStar> DBPassword shut_them_@11_d0wn </App>

    This will give you a different value for DBPassword depending on the current setting of App. There are other modules that do this too, and of course a simple Perl config does this beautifully.

    Where do the smarts about concatenating paths based on projects go? In your config class, which will look at the parsed config data and figure out the answer when you call a method that needs it.

    I don't think your session question makes sense here. Sessions are not config data. There's no reason they should use the same mechanism. I suggest making that a separate question with more information. I can't tell what the problem is from what you've said so far.

      I'm pretty sold on the OO approach, but there are a few additional wrinkles I neglected to mention. I agree that having a perl-based config file is very handy and easy to use. When we do so, we typically want to use info from the machine config inside the app-specific config file. So...

      # In my cgi script... use My::Config; my $config = My::Config->new(project => 'GuestBook'); require $config->get('config_path') . "app_config.pl"; # Now, inside app_config.pl, what's the best way to get at the config # object if I want to use some of the methods? # Just create another new object? It seems wrong to do it again, # but I suppose the overhead # is probably trivial. # Inside app_config.pl package My::App; use My::Config; our $config = My::Config->new(project => 'GuestBook'); if ( $config->get('env_dev') ){ # We're in dev, so set the dev passwords. } # etc.

      Now, how do you feel about then using the $config object as a package variable to access it in other code that is part of the framework but outside the scope of the original script? Using the code above, I could then do this, right?

      package My::App::Page; my $db_password = $My::App::config->get('db_password');
        I would use a class method rather than a package variable. No need to expose that.

        For handling the application scoped and machine scoped configs, I would make it possible to access both through a single object. You might make an application config class that subclasses the machine config class, or you might just have a class that always reads the machine config file in addition to the app config file and merges them internally (i.e. there is no API for accessing the machine config outside of an application context).

Re: Configuration Best Practices for Web Apps
by clinton (Priest) on Jan 24, 2006 at 18:40 UTC
    I have a couple of sites which are configuration heavy, and I've written a module which has made life really easy for me.

    I would welcome feedback on the module itself (included below - see readmore)

    This covers the application and machine specific configuration mentioned above.

    All config is kept in nested YAML files (YAML is a REALLY easy way to express complex Perl variables simply). These nested files form a tree of keys. But you can also have trees within each file itself, so a file may look like:

    /configdir/global.conf: site: name: My website name url: http://mywebsite.com langs: en fr es handlers: login: My::Login home: My::Home etc
    So you can could say something like :
    print C('global.site.handlers.login')
    Alternatively, you could have the directory structure:
    /configdir/global.conf /configdir/global/site/handlers.conf
    or a combination of the two. The sub-directories are read first, then the config files, so that the handlers info in global.conf would be added to the handlers info already read from handlers.conf

    Then, to handle machine specific config, you can have a file only on that machine called local.conf (at any level in the directory structure) in which you specify particular keys that you would like to override. For instance:

    /configdir/global/site/local.conf: handlers: login: New::Login
    would only override the handler for login.

    I can think of a number of improvements, such as being able to specify different config dirs depending on the application, but it does what it says it does.

    You can even reload the config while the website is running, but the config would no longer be shared between processes and you would need to reload the config for each apache process.

    Like I say, I would appreciate feedback on this - anything stupid that I've done, ways to improve it etc. It uses the new YAML module YAML::Syck which is a lot faster than the pure Perl version, and also handles UTF8.

Re: Configuration Best Practices for Web Apps
by dragonchild (Archbishop) on Jan 24, 2006 at 17:08 UTC
Re: Configuration Best Practices for Web Apps
by thedoe (Monk) on Jan 24, 2006 at 17:30 UTC
    • be able to load shared config info that is machine-specific but relevant across many apps (e.g., directory prefixes, current environment (dev, prod), current server name, etc.). Would like to be able to pass in a project name to allow some automatic customization of directory paths
    • For this, why not create config object, similar to what you are already doing. However, instead of trying to pass the project name in during the use My::LocalConfig qw(project_name);, do something like this:

      use My::LocalConfig; my $local_config = new My::LocalConfig '<project>';
      Then, within your new method, you can customize some of the directory paths.

    • be able to load app-specific info, and use variables from the shared info to do it (e.g., use a path from the shared info to tell me where to find my app-specific config file on a given system)
    • Following along the same path as before, we can now use an object to clarify app-specific info. Since information from the machine config is necessary, it can again be passed into the new method of your app-specific config object.

      require "$local_config->{CONFIG_PATH}/AppConfig.pm"; my $app_config = new AppConfig $local_config;

    • load 'session scoped' info appropriately with a layer of abstraction so apps aren't directly using $ENV{REMOTE_USER}; I want a layer because depending on the auth system we use, the current username might be in REMOTE_USER or SOME_OTHER_ENV and I'd like to abstract that from direct use by apps
    • For session information I would suggest using a module such as CGI::Session or Apache::Session. I have used CGI::Session before and found it quite nice, but never under a mod_perl environment. The documentation for CGI::Session did not mention anything about mod_perl, but the documentation for Apache::Session says the following: "Apache::Session is designed to work with Apache and mod_perl, but it should work under CGI and other web servers, and it also works outside of a web server altogether."

    I hope this helps in your search for configuration options.