kurt_kober has asked for the wisdom of the Perl Monks concerning the following question:

Howdy Monks,

I work with a system that generates a steady flow of outbound files. These files need post-processing depending on who they're being sent to, and this is being done via Perl. Some files need only to be moved to a particular location, but others need to be renamed based on their contents, and some need to be verified against an Oracle database and have their contents modified. Some require all these mods and more to come.

Point is: originally I wanted the script to load the necessary post processes (extensions) on the fly. (i.e. Why should it load DBI and Oracle::DBD if all it's doing is moving a file?) I achieved this by writing separate packages for each necessary routine and loading them with a require statement. The packages would all have a subroutine called "process" that took the identical argument, a hash ref containing the list of files to modify (and some other parameters).

Example:

If the outbound files were for PartnerA, the program would read a little config file and see that PartnerA needed the following extensions: modify, rename, dbread, and move--in that order. Those extensions would be pushed into an array @ext: 

foreach $ext (@ext) { 
   eval "require $ext" 
   $args_ref=$ext->process($args_ref)
}

[download]

The above is written. It works. And yet, it's difficult to maintain and I wonder if my initial aversion to loading modules I'll never use was worth the current price of maintenance.

And so I pose a question to the monks: is the above a reasonable idea, or is there a better way, like perhaps a simple chain of separate Perl programs rather than dynamically-loaded modules? The only requirement is that there be one and only one central "controller" script that launches the other processes. I can't build a bunch of different scanners for different partners--we have too many.

Many thanks,
kurt_kober

Replies are listed 'Best First'.
Re: Is it wise to dynamically load modules?
by Abigail-II (Bishop) on Oct 14, 2003 at 16:41 UTC 
    foreach $ext (@ext) { 
   eval "require $ext" 
   $args_ref=$ext->process($args_ref)
}

    [download]

    The above is written. It works. And yet, it's difficult to maintain.

    Well, that's a pretty simple loop, and it didn't take you many lines to describe what it does, and why. Yet you claim it's difficult to maintain, so I expect there's something else going on as well. Why is it difficult to maintain, and why do you think it's easier if you put the functionality in different programs, and chain them? You will still be chaining, and you still need to logic to determine what to chain, and in which order.

    Abigail

[reply]
[d/l]
      The difficulty of maintenance came from instances where a weakness in one link of the chain would lead to problems when I was developing a new link. So there were times when it became confusing to know exactly where a problem was occurring and how to fix it. Plus, changes to the process would sometimes mean changes to all links--of course this would be a problem whether I was chaining subs or separate scripts.

      Guess the main point of the question is whether requiring the modules on the fly was an accepted practice or something I should avoid. Since nobody has yet recoiled in horror, I think I'll stick with it.

      Thanks!

[reply]
        Requiring modules on the fly is a common technique.

        Abigail

[reply]
Re: Is it wise to dynamically load modules?
by dragonchild (Archbishop) on Oct 14, 2003 at 16:55 UTC
    A few comments:
    1. Did you make the decision to not load unused functionality based on Benchmarking or was it an ephemeral "I don't wanna"? This sounds like premature optimization, to me ...
    2. Abigail-II is right - the system you have sounds very maintainable ... it's well factored - each extension does one thing, only one thing, and (presumably) does it well ... What's the problem?
    3. I have written something very similar to this for one of my current clients (with a little more pizazz). I load all my modules and it just runs. Optimization comes later, when SLA's are being violated ...
    4. If you absolutely need more performance, why not eliminate the extra Perl interpreter every run and make this into a web-type deal using Apache/mod_perl? You have a script that calls a CGI through port 80 ... that would be optimized ...

    ------
    We are the carpenters and bricklayers of the Information Age.

    The idea is a little like C++ templates, except not quite so brain-meltingly complicated. -- TheDamian, Exegesis 6

    ... strings and arrays will suffice. As they are easily available as native data types in any sane language, ... - blokhead, speaking on evolutionary algorithms

    Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.

[reply]
      1. It was the ephemeral "I don't wanna" combined with a dose of "Gee it sure would be clever if I could do this".
      2. See my answer to Abigail. I may have--ahem--overstated the difficulty of maintenance somewhat.
      4. I would do that, except the environment isn't running Apache, I'm leaving my current assignment in three weeks, and I'm way too dumb to install Apache and learn CGI in that amount of time. :)

      Thanks very much!

[reply]
Re: Is it wise to dynamically load modules?
by perrin (Chancellor) on Oct 14, 2003 at 19:03 UTC
    Two tips for you:

    • You should check the return of eval, as well as $@, to see if there was a problem loading the module.
    • There are modules like autouse that do all of this for you, if you'd rather not bother doing it yourself. The standard AutoLoader module is very similar, and can be used to do this (it loads subs when they are requested, and those subs could load other modules).
[reply]

Back to Seekers of Perl Wisdom