ph713 has asked for the wisdom of the Perl Monks concerning the following question:

What would you do in order to code and deploy a complex, large package of perl code to a vast array of widely varying *nix machines?

The details (excuse the rambling, I'm still figuring this out as I type it):

I'm getting to the latter stages of writing a large project in perl (well, not too large, but on the order of ~10k lines of original code). So here I am with a baseline requisite of perl 5.6.1 existing somewhere on the target machine (in a finite list of possible locations) as my only enforceable requisite. The machines number in the thousands, and there are several different *nix operating systems and cpu manufacturers involved.

The software package I'm deploying already requires a number of Pure Perl modules from CPAN even on 5.8.x, so I include them with the software itself. I don't use any XS modules that aren't in 5.6.x core, because I can't deploy them easily across the various architectures (just because perl happens to be installed doesn't mean that I can build XS modules - perl may have been built far away in space and time where some functional C compiler once existed).

And now that I'm reading Perl Best Practices (!! where was this book a year ago when I started this project ?? :) ), I have a laundry list of new modules I want to use as well, in order to promote more maintainable code.

On to the specific questions:

1) Given that there may be multiple "perl" executables installed somewhere within a finite list of potential "bin" directories, which may or may not be in $PATH in some order, how do I make sure my deployed package executes using only the best available perl? I have a hackish solution (basically, something that even executes under perl5.0 and can search for other perls, examine "perl -v", and re-execute with the best version found), but seeing as the re-execution has to happen fairly early on, and that the initial "perl" which executes the script could be ancient, I pretty much have to stab this code manually into the BEGIN block of scripts, and the application has several seperate scripts and daemons which can be run from the commandline independantly.

2) It's easy enough for me to include a copy of a bunch of CPAN pure perl modules in my private application modules directory to fill in the "gaps" for whatever isn't part of the core set or doesn't happen to be installed via the OS on whatever machine. But how can I cleanly go about making my scripts/modules use the "best available version" between what I supplied and what's available locally? If I ship SuperModule.pm v1.21, and the machine has v1.23, or vice versa, how can I always end up with the latest and greatest?

3) What's the proper way to get your @INC set up for a custom directory for a given application? For example, currently my software package installs under "/opt/bamf" (names changed to protect the innocent), with user-executable scripts in /opt/bamf/bin, executable daemons in /opt/bamf/sbin (some of which are started from an init script, let's not even go there with all the different *nix standards for init scripts), and a tree of perl modules (some from CPAN per above, some custom to this application) under /opt/bamf/lib (like /opt/bamf/lib/Widget/Maker.pm). Of course "/opt/bamf" shouldn't be a hardcoded location, so for now I've made it the "default", and let be overridden by an environment variable, but it's ugly. Every script contains this line very near the top:

BEGIN{$ENV{BAMFPATH}||='/opt/bamf';push(@INC,$ENV{BAMFPATH}.'/lib');}
It's ugly, and I feel bad even writing it. Anyone have a better approach to all this mess?

Replies are listed 'Best First'.
Re: Deployment Qs
by maverick (Curate) on Oct 06, 2005 at 22:18 UTC
    I'm acutally in a similar boat with a project that I'm working on. Here's some things that can help.

    1) if you're using ye-olde "perl Makefile.pl" method of installing your system, or you *can* use this method; add options to your WriteMakeFile configuration that look like this:

    INSTALLSITELIB => '/opt/bamf/lib', INSTALLSCRIPT => '/opt/bamf/bin', EXE_FILES => ['relative/path/to/pl1','relative/path/to/pl2]

    The first line (I think) will override the default location for the installation of the pm's. The second spells out where the executable scripts go, the third lists the executables to be installed. The handy thing that this process also does is that it fixes the #!/perl/lives/here lines in those scripts to match the location of the perl that executed the Makefile.PL.

    2) You could use CPAN to install / update each system to the latest version of the required modules at the time you run 'perl Makefile.pl'...this of course assumes that CPAN is configured on each of these systems.

    3) If it's a rule that your application's layout has 'bin' and 'lib' as siblings of the same directory, you can use File::Spec to figure out where lib is at run time or within a BEGIN block.

    use File::Spec; my $lib_path = File::Spec->rel2abs($0); $lib_path =~ s/\/bin\/[\w\d-]+\.pl/lib/; push(@INC,$lib_path);

    Another nice thing about using the Makefile.pl approach is that it will run your test suite before it performs an upgrade of your application. This way if some new feature doesn't work on machine X, you didn't trample an old (but working!) installation.

    edit fixed typos.

    /\/\averick

      I went out and saw a really stupid cheesy movie with someone, and it got my mind off of this project (for the first time in a few weeks I think) long enough for some subconscious something to kick in. After getting home I was immediately aware that avoiding the make process was the mistake that led to this mess. It just took some time away from it for it to sink in after you mentioned it. My path is clear now, I need to:

      1. Make a simple /bin/sh script (lets call it Installer.sh) that will run on any remotely reasonable *nix system and detect the latest available perl from the finite list of possibly directory paths. Then it executes something akin to "/some/where/perl Makefile.PL && make && make test && make INSTALLDIR=$1 install" with the appropriate perl. This isn't hard to do by sticking to pure bourne shell constructs and simple tools like basic grep.
      2. Get a perl-style Makefile.PL built up for this project, and customize it so that it transforms #!/usr/bin/perl on the scripts to the perl that was used during the make, and so that it can be told what directory to install the software to, and transform some push(@INC,'...') line in all the scripts to point at a library directory located there, like what you described
      3. Profit! (or at least, keep my job :) )

      Anyways, thanks, your mentioning of using the standard perl Makefile.PL stuff and those variables got me thinking straight(er).

        IIRC you wont need to do step 2. If you scripts are identified as such then the install process will do this for you.

        ---
        $world=~s/war/peace/g

      Unfortunately I'm not using perl Makefiles for this, at least not yet. I had to some degree pretty much dismissed the perl make system for this project, but I may need to go back and re-examine that decision. Ultimately, this project is not a perl script or a perl module - it's a complete software package that runs multiple daemons as root on the system and offers a few commandline tools for sysadmins as well (it's essentially the client side of a systems management/monitoring system of sorts, with some rather hardcore requirements in terms of scalability and precision).

      I was doing something very similar to your File::Spec method earlier on, and it may ultimately be what I have to do in production. However, I was doing it "manually" without File::Spec, which resulted in ~8 lines of code instead of your 2-3 (it's significant to me, because that block of code basically has to be copy+pasted into every script in this system, there's no real sane way to modularize it and eliminate the redundancy that I've found). Tack on the "find the best perl interpreter" problem, and you've got a decent little block of code which must be pasted into BEGIN{} at the top of every script.

      It looks like File::Spec 0.8 is when rel2abs() was introduced, and File::Spec 0.8 became part of the core in perl 5.5.x somewhere, so it should be cool for me to use it implicitly in the begin block like that. At least that shortens things up a bit

Re: Deployment Qs
by BUU (Prior) on Oct 06, 2005 at 21:23 UTC
    I don't have to time to go in to a lot of details at the moment, but have you considered PAR? It allows you to bundle your script(s), Perl, and any necessary modules, in an easy and convient executable.
      I passed over PAR earlier in this process. For one, I don't really want to push out a seperate perl executable to the majority of the machines who could do fine with their own local copy, and secondarily, I'm pushing to many incompatible architectures, all of which would need their own build of perl. I'd prefer not to get into enumerating and having custom perl builds for every possible arch. (Currently, the perl code itself has been kept platform neutral where at all possible, and tested on a least a good subset of the potential architectures).
Re: Deployment Qs
by Moron (Curate) on Oct 07, 2005 at 14:01 UTC
    1) I would identify through testing the absolute earliest version your code can tolerate. But then I would investigate (by testing against even earlier versions) what the impact would be of modifying code to push that limit backwards to allow as many sites as possible to be able to use the suite in question and make a feasibility decision as to what the earliest version should be.

    2) A 'require ' + version (e.g. require 5.004;) prevents a script running in an environment earlier than the earliest version your policy then supports.

    3) Module FindBin::libs offers automated ways to load lib directories into @INC/%INC without any minimised (hard-) coding (you have to tell it where to start recursively searching for modules).

    -M

    Free your mind