Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

putting perl and modules in your source code repository

by perl5ever (Pilgrim)
on Apr 13, 2009 at 19:54 UTC ( [id://757268]=perlquestion: print w/replies, xml ) Need Help??

perl5ever has asked for the wisdom of the Perl Monks concerning the following question:

I'm working on a large perl application that's been around for a while, and one thing we've discovered is that we need to put the source for perl itself and the CPAN modules we are using in our source code repository with our app.

I was wondering if any had any experience or ideas of how to do this so we don't have to rebuild perl and the modules all the time. The set of modules doesn't change very often, but at runtime we want to be as sure as we can that the module tree we are using is as specified in the repository.

Does anyone have any experience with this? Do we commit the module packages (.tar.gz files) into the repo, or do we commit the files from the installed module tree? And what would be a good way to determine if a previous build of perl and modules is compatible with the needs of the current app source code? Thanks!

Replies are listed 'Best First'.
Re: putting perl and modules in your source code repository
by Tanktalus (Canon) on Apr 13, 2009 at 20:17 UTC

    How you're going to do this will largely depend on the infrastructure you have available. For example, we simply installed global perl's in /some_share/$platID, and then have some logic in shell to pick the current platform ID based on uname, etc. However, what we haven't done is automate the building of anything (though I've cried for it).

    What I'd prefer seeing is that the perl source was put into its own repository/branch/whatever that we could build on each platform. And that part of this was that we could insert new CPAN tarballs into the mix, and simply rebuild on each platform and reinstall.

    In the meantime, since I can't easily add modules to the global perl trees, I simply check in the tarballs to the build, and have make run my preparation script which untars them and runs Makefile.PL/make/make install or Build.PL/build/build install as appropriate to install to the build directory (i.e., a private lib directory). But I'd like to be able to submit it globally, and, using a form of continuous integration, get it applied to the global perls automatically.

    As for your last question, "prove" is a good way to determine if a given build of perl is compatible. If the unit tests run clean, it's probably sufficient (assuming adequate unit test coverage). But I'm going to end here before I head into a full-blown rant.

      One thing I've found that it is extremely important to be able to recreate exactly the same environment. This is useful for deployment, debugging, etc. We've run into too many problems where we see a problem in production that we can't reproduce in development. That's why want to store perl and the modules in the repo. We also have a global perl install, and it's causing too many problems. For one thing, you can't add/modify a module without potentially disrupting someone else's work, and then keeping dev, test and prod in sync is a pain.

      Re: prove... that would be helpful if we had any unit tests. :-)

Re: putting perl and modules in your source code repository
by eyepopslikeamosquito (Archbishop) on Apr 14, 2009 at 06:46 UTC

    We run the identical version of Perl with an identical set of CPAN modules on all our many different Unix boxes (multiple versions of: AIX, HP-UX, Solaris, Red Hat Enterprise Linux (RHEL), Digital UNIX, Tru64 UNIX, IRIX, UnixWare, SCO Unix, ...).

    I install this perl in the same well-known location on each of our boxes and adjust the PATH so that we use this version of Perl for all our work. We never use, or meddle with, the system /usr/bin/perl. We also ship our Perl distribution with all our products. Using the same (stable) Perl distribution everywhere avoids many annoying portability glitches ("it's easier to port a shell than a shell script").

    To achieve this, I put the source tarballs for perl and all the CPAN modules on a shared drive available to all our Unix boxes, then run a script that builds perl from source (Configure, make, make test, make install), followed by building all the CPAN modules from source (perl Makefile.PL, make, make test, make install). The script contains an array of CPAN modules and builds them in the specified order. Because our Perl distribution must run on customer machines, and we don't want to meddle with their /usr/lib or /usr/local/lib, I have to take care to build any CPAN modules that depend on C libraries (e.g. XML-Parser) in such a way that they are wholly contained in our Perl distribution and don't depend on the contents of /usr/lib or /usr/local/lib. Only our build script is currently under version control. It would be easy to put all the tarballs under version control also -- and I probably should -- though I haven't bothered so far, mainly because they are essentially unchanged from the CPAN tarballs.

    This scheme has worked reasonably well here because we require stability and repeatability (we typically update our Perl distribution only once every two years or so). One nuisance is when we need to make an emergency fix to a CPAN module. In that rare event, I unpack the CPAN module tarball, make the fix, then repack it into a new tarball with a different name, updating our build script to use the new tarball. Ugly. Since I've only needed to do this once, I haven't investigated a better way to deal with this use case. Ideas welcome. Notice that if you need to make an emergency fix like this, it's in your best interests to send the fix back to the module author as soon as possible, so as to get a proper code review and to avoid the long term burden of maintaining your own custom fork.

    Though the above scheme has worked well for me here, it's probably not appropriate in a more volatile Perl environment, and I'm interested to hear how other folks deal with that.

    BTW, so as not to meddle with customer /usr/lib or /usr/local/lib (and to avoid the dreaded LD_LIBRARY_PATH, see also Re: Portable Perl?), and to not require root permissions, our install script used to binary-edit our executables, replacing their artificially long build PATH with the customer chosen install directory with our lib sub-directory appended.

    Related CPAN Modules

    Pinto References

    Related Perl Monks Nodes

    • Re: Out-of-the box Perl version - lowest common denominator by me (2021) - notes that non-technical customers lack the skills and desire to build their own perl ... while building our own perl vastly improves and simplifies customer support, e.g. it's easy to set up test environments in our office identical to what the customer is using
    • Howto Bundle Perl (2009) - asks for advice on bundling Solaris perl with a software product that depends on it

    References Added Later

    Updated: "Related CPAN Modules" and following sections were added long after the original reply was made.

Re: putting perl and modules in your source code repository
by cutlass2006 (Pilgrim) on Apr 14, 2009 at 08:43 UTC

    I have to do a similar thing.

    What works for me is that I use subversion svn:externals tag to create a link to CPAN source control version of desired package then build that whenever I need to deploy as part of my deployment process.

    ps: this only works if there is a repository available in the first place, but I seem to have always found one for code I am interested in

Re: putting perl and modules in your source code repository
by admiral_grinder (Pilgrim) on Apr 14, 2009 at 18:30 UTC

    We standardize on a perl verison/build and build CPAN modules against it. We will test it on our boxes first and make patches if needed. When it works we will save off the blib directory from the built module, copy it into our custom module, and store it in our repo (remember to turn off keyword expansion or any other file changing operations in your repo). Then we use a special Makefile.pl that only installs the contents the blib directory. The only conflicts we see is the .exists files.

    If you want to keep the original source around you can stuff the tar.gz files into the repo. We don't do that and will go back out to CPAN for the versions we need. Few times we couldn't find a older one, but found out that the new version was better for us anyways.

    And what would be a good way to determine if a previous build of perl and modules is compatible with the needs of the current app source code?"

    Testing. However, unless there is some major changes to the module, then you won't find too many issues.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://757268]
Approved by toolic
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others sharing their wisdom with the Monastery: (9)
As of 2024-03-28 12:24 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found