In a nutshell: how do you deliver a Perl-based utility that just works "out of the box" for most users, even those that don't have the ability to upgrade their Perl version or install new modules?

I work on Cons, a Perl-script software construction tool (i.e., Make substitute) that was recently added to the stable of GNU programs. We try to make it as easy as possible for people to try Cons, so we put a lot of effort into making sure it works as nearly "out of the box" as possible. In other words, it should look like a stand-alone utility, (almost) regardless of what Perl version they're running or whether they have permission to install modules. Having to install a lot of new modules, or a new version of Perl, would turn away people that we'd like to encourage to to test-drive the utility. (Yes, non-administrators could install modules in their own directories and fiddle with -I options or @INC, but even with the CPAN module, that starts taking things too far into the realm of Perl Gurudom for many people. Sad, but true.)

Accomodating different Perl versions is relatively easy. As far as the coding itself is concerned, this just means using lowest-common denominator idioms that work in all versions we support: no "foreach my $var" (not introduced until 5.004), no "\z" in regexes (not until 5.005), etc. We use an extensive regression test suite to make sure Cons works under Perl versions back to at least 5.003 (the earliest version I can test).

Modules are a stickier issue. In some cases, the lowest-common-denominator approach works: just use module interfaces that date back to 5.003 Core Modules. But in other cases, a feature we add requires using a newer version of a module, or a module that wasn't in the Core in an earlier Perl version. For example, portable multi-volume support in file names requires File::Spec 0.8, which wasn't available until Perl 5.6.0--and even then, File::Spec 0.8 used "\z" in regexes, so the module itself doesn't work with pre-5.005 Perl.

One solution would be simply to not rely on these (or any?) modules and code everything on our own. But this loses big-time because: it reinvents the wheel; it introduces potential errors; and it risks perpetuating "cargo cult" code if someone else cuts-and-pastes the reinvented code (as someone is almost sure to do). So what to do?

Our current middle-ground solution is to use the real module's interface, but ship an internal version of the module inside the script--for example, our own "package File::Spec" inside Cons. These "internal modules" contain mostly cut-and-pasted methods from the real modules, with as few fixes/changes/home-brew code as we can get away with. And, almost most importantly, they have big comments that warn people to not cut-and-paste this internal code, but to use the real modules.

The main advantage here is that we still use the correct, up-to-date module interfaces. Consequently, anyone looking at the rest of the code still sees the "right" way to use File::Spec and the other modules. But most importantly, in the future we can get rid of the "internal" modules whenever we drop support for 5.003, 5.004, etc., and can rely on those interfaces we use being part of the the Core Modules in whatever Perl versions we'll still support.

I thought this was worth a meditation not because this is necessarily the most brilliant solution, but because of how this set of requirements--delivering a Perl-based "product" that works out-of-the-box the way a precompiled binary would--changed the way we had to think about incorporating Perl modules. This solution still has drawbacks, but it's the best we've come up with to try to deliver a utility that satisfies the requirement and just happens to be written in Perl. (Of course, if a more enlighted monk has a better way, I'd be eager to hear about it...)

Replies are listed 'Best First'.
RE: modules out of the box
by merlyn (Sage) on Sep 05, 2000 at 18:25 UTC
    Please don't include anything in your distribution that uses a CPAN-registered module name. I've been bitten more than once on that, the most memorable being the "Minivend" shopping cart package.

    The "Minivend" author started with the attitude you had: he had wanted to make a "turnkey" solution for end users, and needed some modules that perhaps were not commonly installed, so he had included frozen versions of them into his distribution, under the original names.

    Well, time passed, and he didn't incorporate the latest version of those modules into his distribution, but it came time for me to install Minivend to test it to see if I wanted to do a column about it. Imagine my horror when Minivend uninstalled my more modern versions of those handful of central modules, and replaced them with his own downrev versions!

    I immediately posted a note to his support list and to P5P warning people that "minivend is evil", and got some interesting responses, but mostly the P5P group sided with me: a distribution should not have private versions of public modules under the CPAN-registered name. This leaves him (and you) with two solutions:

    1. Include (as you suggested) a version of the module, but safely tucked away under your own namespace. Like a version of Minivend::Foo::Bar for a module that everyone else knows (or once knew) as Foo::Bar.
    2. Simply list the prerequisites in the Makefile.PL, and let the installer sort it out.
    The latter is highly preferred, although the former insulates you from any "upward compatibility" problems, but most modules I've seen only get better, and have sacrificed very few items in the name of backward compatibility.

    The "Minivend" author learned his lesson, and now does the latter. I suggest you seriously look at doing the same. Do you really think a site can be self supporting if they can't even install a module or two from the CPAN? There is a support cost to using Open Source software, and part of it includes being able to include components. If they want Microsoft, they know where to get it. {grin}

    -- Randal L. Schwartz, Perl hacker

[reply]
      Good point. I didn't explain one thing clearly, though...

      We don't actually ship our own, separate Spec.pm file (e.g.). Since Cons is a single script, we just "package File::Spec" within the script itself, and thus don't go outside to look for any module. I think this sidesteps the issue of causing trouble for anyone else's module installation, but let me know if I'm overlooking something.

      Do you really think a site can be self supporting if they can't even install a module or two from the CPAN?

      No, but that's not what we're trying to solve. We're trying to make it easy for someone to pull down the tool and experiment with it, regardless of how much their IT department has their system locked down. During the initial experimentation phase, every barrier they run into becomes a point at which they can say, "I'm not getting anywhere with this," and go do something else. We're trying to remove as many of those barriers as possible, so that any programmer can try Cons with a minimum of fuss, see how cool it is, and then figure out how to get their department to use it for real...
[reply]
        Then one solution would be to include Spec.pm with your distribution, and have your Makefile.PL install it as Cons::Spec (or whatever namespace the CPAN registrars gave you) in the absence of a detected suitable File::Spec. You can load the code in your modules like this: 
        BEGIN {
  for (qw(Cons::Spec File::Spec)) {
    eval "use $_"; last unless $@;
  }
  die "Cannot find Cons::Spec or File::Spec" if $@;
}

        [download]
        A little bit of runtime launch cost can provide a ton of flexibility for your installers. If you want, you can even edit this at compile time into a simple use File::Spec, and avoid the runtime cost.

        -- Randal L. Schwartz, Perl hacker

[reply]
[d/l]
RE: modules out of the box
by elwarren (Priest) on Sep 08, 2000 at 00:52 UTC

    There is a perl script out on the net that will look at all the code that you include in your script and builds a single file out of them all. This one file can then be distributed without any additional files. I strongly doubt it would work on anything that requires actual compiling and linking (like the DBI modules.) While not the best solution to your situation it would make distributing your code easier. You could continue to use the modules of your choice while developing, and just generate the single perl file during your make process.

    I think I read about the script in an old perl journal, I'll try looking for it tonight...

    Another option, and def not my choice, would be to distribute perl with your code. Users could choose between a full install or expert install. The expert install would let the user use their own perl installation, while the full install would install a base version of perl with only the required modules. This private version would only be used to run your program. This would take a good deal of work for you to setup initially, but would allow you to control exactly what they have. I think that win32 users would be the only ones to accept this, they already do with ActiveState and some nt web servers include their own also.

    This also opens up the problem of maintaining several different compiles for several different platforms. Not fun.

    The mailman web email script was distributed as a windows file instead of a unix file and was always a pain in the ass to install on our server. Please don't do anything quite that silly.

[reply]
RE: modules out of the box
by mischief (Hermit) on Sep 06, 2000 at 16:33 UTC
    Perhaps a solution would be to compile your program as an executable? You could distribute the source with the binary and just document the fact that it uses modules which have to be present if the user wants to recompile with changes. It would work out of the box, but for those that want to tinker, the source is there.
[reply]
      I think a good option would be to copy the modules you need into your namespace, and prepend any reference to them with your namespace in the code (clean option), or to prepend the path to your namespace to your @INC at runtime, so your modules will be loaded first (hack, not recommended).

      Just as an aside, in Germany there is a very successful commercial content management system called imperia implemented in Perl that does just this to include its share of modules from the CPAN.

      Christian Lemburg
      Brainbench MVP for Perl
      http://www.brainbench.com

[reply]

Back to Meditations