Building perl for appliances

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi, would appreciate your help on this.

We're building a Linux-based appliance with perl, all the software (linux kernel, perl, applications) are build from source, into an staging area, then packaged together. We managed to get perl into the build process. My question is how to include extra packages (e.g., DBI, Crypt, etc.)

(1) When/where to include a specific package (XS extensions)? I can add the package source into the source "ext" directory, and build as a part of the base perl; or adding later when the base perl has been built (the usual Makefile.pl/make/ processes.

(2) The second question is how to resolve all dependencies, do I need to add one package at a time, i.e., download the source, put into the build, and figure out what it depends on. When I use cpan, some of these dependencies can create lots of other dependencies, I certainly don't want them all. Is there an easier way?

Thanks.

Comment on Building perl for appliances

Replies are listed 'Best First'.
Re: Building perl for appliances by tsee (Curate) on Nov 20, 2009 at 13:55 UTC
Let me start with answering question (2). You say that if you want to include package Foo in your distribution and Foo depends on Bar1, Bar2, Bar3, then you don't actually want to include Bar* in your distribution as well ("certainly don't want them all")? That doesn't make any sense to me. Foo declared the dependency with a reason. But maybe I'm misunderstanding you. In Perl, there is essentially three types of library directories: core-, vendor-, and site lib. They are intended for use by the perl core, the vendor and the site admininstrator respectively. Since you intend to create a binary package of perl, you're the vendor. I suggest you use "cpan" to install your custom packages into the vendor directory and ship them as part of the package. On some level, I actually like the idea of including extra packages in the base perl distro and then compiling them as part of the core installation. There's a couple of things that can cause trouble with this: It's manual and thus tedious and error prone. In current stable perl releases, there's a bunch of magic scripts and MANIFEST that need updating with the extra files. The module sources (Makefile.PL/Build.PL) may need tweaking so they don't install into the vendor library directories. You have to be careful not to ship such a beast as "perl 5.10.1" because it's not. It's perl 5.10.1 PLUS your libraries. While I'm nitpicking a little, neglecting such distinctions has the potential to cause a lot pain. In the current development branch of perl, the directory structure has been reorganized to make maintenance of the modules that live both in core and on CPAN easier. Where there used to be a giant lib/ directory and an ext/ directory for XS modules, there is now: lib with the few modules that are needed super-early in the build process. ext with subdirectories each containing a CPAN-distribution like setup of really-tied-to-perl-core modules dist with all the modules that live primarily in core but have a CPAN version published. (Changes go to perl core first and get pushed out to CPAN as appropriate.) cpan with all the modules that are maintained primarily on CPAN and that are simply updated in core ext, dist, and cpan are organized in subdirectories which mimick an extracted CPAN tarball as closely as possible (usually ignoring the Makefile.PL). Lots of hardcoded information has been eliminated and missing Makefile.PL's are usually auto-generated. Just extracting a CPAN distribution is not likely to just work yet but the amount of fuzzing to get it included should be pretty low nowadays. Note that there is no stable release of perl that features this reorganization. You can experiment with the recent 5.11.X development releases. Cheers Steffen	[reply]

Replies are listed 'Best First'.

Re: Building perl for appliances
by tsee (Curate) on Nov 20, 2009 at 13:55 UTC

Let me start with answering question (2). You say that if you want to include package Foo in your distribution and Foo depends on Bar1, Bar2, Bar3, then you don't actually want to include Bar* in your distribution as well ("certainly don't want them all")? That doesn't make any sense to me. Foo declared the dependency with a reason. But maybe I'm misunderstanding you.
In Perl, there is essentially three types of library directories: core-, vendor-, and site lib. They are intended for use by the perl core, the vendor and the site admininstrator respectively. Since you intend to create a binary package of perl, you're the vendor. I suggest you use "cpan" to install your custom packages into the vendor directory and ship them as part of the package.

On some level, I actually like the idea of including extra packages in the base perl distro and then compiling them as part of the core installation. There's a couple of things that can cause trouble with this:

It's manual and thus tedious and error prone. In current stable perl releases, there's a bunch of magic scripts and MANIFEST that need updating with the extra files.
The module sources (Makefile.PL/Build.PL) may need tweaking so they don't install into the vendor library directories.
You have to be careful not to ship such a beast as "perl 5.10.1" because it's not. It's perl 5.10.1 PLUS your libraries. While I'm nitpicking a little, neglecting such distinctions has the potential to cause a lot pain.

In the current development branch of perl, the directory structure has been reorganized to make maintenance of the modules that live both in core and on CPAN easier. Where there used to be a giant lib/ directory and an ext/ directory for XS modules, there is now:

lib with the few modules that are needed super-early in the build process.
ext with subdirectories each containing a CPAN-distribution like setup of really-tied-to-perl-core modules
dist with all the modules that live primarily in core but have a CPAN version published. (Changes go to perl core first and get pushed out to CPAN as appropriate.)
cpan with all the modules that are maintained primarily on CPAN and that are simply updated in core

ext, dist, and cpan are organized in subdirectories which mimick an extracted CPAN tarball as closely as possible (usually ignoring the Makefile.PL). Lots of hardcoded information has been eliminated and missing Makefile.PL's are usually auto-generated. Just extracting a CPAN distribution is not likely to just work yet but the amount of fuzzing to get it included should be pretty low nowadays. Note that there is no stable release of perl that features this reorganization. You can experiment with the recent 5.11.X development releases.

Cheers
Steffen

[reply]