Mutant has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I'm currently working on an application that has an interface to create a configuration for various projects, and then a set of libraries to read that configuration for each project. Each project needs to be extremely reliable, and preferably as effecient as possible. The configuration consists of a set of complex data structures.

To achieve this, I'm currently using Storable to write the config to disk, then reading it from each project. This was chosen over a database driven approach for reliabilty reasons (some sort of replication is probably a little too complicated for this application).

I'm concerned, however, that this approach may be inefficient. The application is likely to generate high traffic, so caching the config is probably not a real option (it's running under mod_perl, so memory usage could well become an issue), although the config is unlikely to get updated very often.

Does anyone have any other solutions that will still be as reliable, but possibly more efficient?

Replies are listed 'Best First'.
Re: Configuration Files
by dragonchild (Archbishop) on Nov 02, 2004 at 16:17 UTC
    If you're running under mod_perl and the configuration will never be updated while the site is up, then load it in your startup.pl and every child will share the config data. Save CPU just like you want. What's the problem?

    Being right, does not endow the right to be rude; politeness costs nothing.
    Being unknowing, is not the same as being stupid.
    Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
    Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

[reply]

      Perhaps I didn't explain the problem properly.. The config will be updated (just not very often), and the goal of the project is to make manual editing of config files (including Apache config) more or less non-existant. So I have an application that generates these config files, which then get read on the other side.

      I could generate them in a way so that they're written as a set of variables, and read in to Apache, but there are going to be a lot (about 20+ a week are created), and there's quite a lot of data in each, so I'm concerned that the memory usage would be too great (hence my current solution).

      I hope that makes it a little clearer..

[reply]
        Let me paraphrase what I hear you saying:
        1. I've got a problem that says "Use a database to solve me"
        2. I don't want to use a database
        3. Any suggestions?

        Seriously, you really want to use a lightweight database solution for this. Something like MySQL, SQLite, or BerkleyDB.

        Being right, does not endow the right to be rude; politeness costs nothing.
        Being unknowing, is not the same as being stupid.
        Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
        Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

[reply]
Re: Configuration Files
by samtregar (Abbot) on Nov 02, 2004 at 17:37 UTC
    I'm guessing that you mean something different when you say 'configuration' than what most people do. When I say 'configuration' I'm talking about something:

    1. Small. The largest configuration file I've ever needed had around 50 values organized in 10 blocks. Apache configs usually come in around this size if you slice out all the defaults, for example.
    2. Human readable and editable. Generally in an Apache-esque format using Config::ApacheFormat.
    3. Not Performance Critical. Typically I'll load configuration data at Apache start-up and forget about it. It's just a hand-full of scalars and hashes, how bad could it be?

    It sounds to me like you're dealing with a beast of a different stripe. You're storing data in a non-readable format and you're worried about memory size and access speed. So what are we really talking about? And what's all this noise about being more reliable than the database?

    -sam

[reply]

      Yeah, fair point, I am talking about something different than a 'usual' configuration. What I'm working on is a generator of applications, each with a configuration stored in complex data structures. Maybe 'config's not even the correct term - it's really 'application data'. Human readable is not necessary (and possibly not even desirable) since the config is controlled by the meta-application.

      As for reliability, see the discussion in the replies above.

[reply]
        Ok, then I'm with the rest of the monks. You need a real database and you need one now. Trying to use Storable to house performance-critical application data is just going to lead to tears. It won't scale and it encourages a level of laziness that will quickly outrun your ability to manage change. It can be a nice way to prototype an application in a hurry but it's just not the right tool for your job.

        -sam

[reply]
Re: Configuration Files
by Taulmarill (Deacon) on Nov 02, 2004 at 16:51 UTC
    although i think SQLite is a goot suggstion since it's smal, fast and has its own CLI-Editor (something storable imho doesn't have) i would suggest YAML for config files.
    YAML is human readable and editable, so you can change you config by simply editing a textfile.

    the fastest solution however, would be loading you config information from startup.pl and restart (SIG HUP) you httpd when config changes.
[reply]
Re: Configuration Files
by zentara (Cardinal) on Nov 03, 2004 at 11:09 UTC
    I havn't tried using it yet, but DBM::Deep looks very interesting.
    I'm not really a human, but I play one on earth. flash japh
[reply]
      That's actually exactly what I was looking for... Thanks.
[reply]

Back to Seekers of Perl Wisdom