Building a package for distribution: How do the pros do it?

hesco has asked for the wisdom of the Perl Monks concerning the following question:

In the spirit of giving this season, I'm making progress on getting some of my work out the door, if not yet ready for prime time, then at least in a distributable form.

Spent some time in the archives reading How to make a CPAN Module Distribution, José's Guide for creating Perl modules, ExtUtils::ModuleMaker and a few distractions along the way.

These all seem to assume a much cleaner development environment than I deal with. I'm a pack rat. I'll throw together half-baked notes, save back sample data or test results, sample code and leave it in my primary work directory. cvs does not seem to mind until I do a cvs add _some_new_file_. But I need a way of easily extracting the files I'm interested in sharing, without destroying my work environment, in a ready to distribute form. I call this: build_distro.pl, on its way to being something more useful. Here is a snippet of the docs.

=head2 build_distro()

build_distro() takes a source directory with a properly
formatted MANIFEST file and produces a tar.gzipped package of the file
+s specified in the MANIFEST file.

=cut
[download]

#!/usr/bin/perl -w
use strict;
use warnings;

# name of tar package to build
my $pkg = "Foo_Bar_0_01.tar";
# expressed as a relative path to directory with MANIFEST
my $basedir = "Foo_Bar_0_01";

=head1 NAME

build_distro.pl - build a tar.gz from a MANIFEST

=head1 VERSION

Version 0.01

=cut

our $VERSION = '0.01';

=head1 SYNOPSIS

This script will accept a package name and a source directory
and use them to build a gzipped tarball for distribution of
only those files and directories not commented out from a
MANIFEST list in the source directory.

To use, at the moment:

        /path/to/build_distro.pl

after hard coding in the values for $pkg and $basedir and
changing the current directory to the path where you want to
package written to.

Longrange the interface is imagined as:

    use Module::BuildDistro;

    my $pkg = Module::BuildDistro->new({ -source => 'Your_New_Module' 
+});
    $pkg->build_distro();

    or, perhaps simply:

    module-build-distro --source=Your_New_Module/

=head1 EXPORT

    build_distro()

=head1 FUNCTIONS

=head2 build_distro()

build_distro() takes a source directory with a properly
formatted MANIFEST file and produces a tar.gzipped package of the file
+s specified in the MANIFEST file.

=cut

sub build_distro {

}

print "Our working directory is: ", `pwd`,"\n";
if(-f $pkg){
  warn "$pkg already exists.";
  print "Shall I delete $pkg (y/n)?";
  my $response = <STDIN>;
  chomp($response);
  if($response eq 'y'){
    unlink($pkg);
    print "I have just deleted $pkg.\n";
  } else {
    die "I can not build new distribution without deleting old one.";
  }
}

open(MANIFEST,"$basedir/MANIFEST") or
        die "Unable to open $basedir/MANIFEST. ?: $?, !: $!";
my $result = `tar -cf $pkg $basedir/MANIFEST`;
  if(!defined($result)){
    warn "Failed to create tar ball: $pkg. ?: $?, !: $!";
  }
FILE: while(<MANIFEST>){
  chomp($_);
  if(/^#/){
    print "Skipping $_.\n";
    next FILE;
  }
  unless ($_ eq ""){
    $result = `tar -rf $pkg $basedir/$_`;
    if(!defined($result)){
      warn "Failed to append $basedir/$_ to the tar ball: $pkg. ?: $?,
+ !: $!";
    }
  }
}

$result = `gzip $pkg`;
if(!defined($result) ){
  warn "Failed to gzip the package file.i  ?: $?, !: $!";
}

1;
[download]

So my questions this fine morning are two fold:

Is my hand rolled solution of this morning duplicative? Surely others have already done this. I'd just as soon let others maintain this if there is such a tool, already.
I'm working to learn to better test and process return values. I haven't yet sorted out this $? and $! thing. Can someone here please give me some guidance on how to use these, please? When do I use one or the other?

Thanks,
-- Hugh

if( $lal && $lol ) { $life++; }

Comment on Building a package for distribution: How do the pros do it? Select or Download Code

Replies are listed 'Best First'.
Re: Building a package for distribution: How do the pros do it? by Joost (Canon) on Dec 26, 2006 at 00:12 UTC
I'm far from a "pro" at making CPAN distro's but doesn't your code do exactly what a `cd your_distro perl Makefile.PL make dist` [download] does? By the way, my most favorite file is MANIFEST.SKIP, which will list any files (regexes) to skip while doing `make manifest` "What should it profit a man, if he should win a flame war, yet lose his cool?"	[reply] [d/l] [select]
Re^2: Building a package for distribution: How do the pros do it? by hesco (Deacon) on Dec 26, 2006 at 01:01 UTC
UPDATE: OK, I went `touch META.yml` into being and then `make dist` ran without errors and produced the intended results. This brings up an entirely new set of questions related to reframing my existing work inside of the PBP guidelines which make this sort of packaging possible. But I guess that deserves its own thread, and perhaps some time contemplating those questions in the midst of simply trying to do it for a while. And to answer Joost's question: "Are you using ExtUtils::MakeMaker or using Module::Build?" `$ module-starter --module=Module::BuildDistro $ cd Module-BuildDistro/ $ perl Makefile.PL $ make dist # <---- this throws erros $ touch META.yml $ make dist # <---- succeeds!` [download] Thanks for the lead! Now to start writing MANIFEST.SKIP files for everything. And now for the original message, already in progress. --------- Perhaps so. I had suspected that someone had needed one of these before I wrote this this morning. So I used Module::Starter::PBP to create a framework from which I could build a module for Module::BuildDistro. Just as an experiment. Here is the error I got when I tried that make dist trick you suggested. `hesco@testbed:~/sb/Module-BuildDistro$ make dist rm -rf Module-BuildDistro-v0.0.3 /usr/bin/perl "-MExtUtils::Manifest=manicopy,maniread" \ -e "manicopy(maniread(),'Module-BuildDistro-v0.0.3', 'best');" mkdir Module-BuildDistro-v0.0.3 -e: META.yml not found at -e line 1 Can't read META.yml: No such file or directory make: *** [create_distdir] Error 2` [download] So now I'm wondering how it is I generate one of these META.yml files this seems dependent on. Any ideas? Reading the MANIFEST says that Module::Starter expected that file to be created by make dist. Reading the perldoc for ExtUtils::Manifest allows me to answer your question: "No, my implementation skips any file not explicitly included in the MANIFEST. While ExtUtils::Manifest excludes only those files explicity listed in the MANIFEST.SKIP file." But other than that, yes, I think this might work for me, just as well, as soon as I work out this META.yml issue. Thanks for the lead. -- Hugh if( $lal && $lol ) { $life++; }	[reply] [d/l] [select]
Re^3: Building a package for distribution: How do the pros do it? by Joost (Canon) on Dec 26, 2006 at 01:09 UTC
Are you using ExtUtils::MakeMaker or using Module::Build? As far as I know, using ExtUtils::MakeMaker (and then running `perl Makefile.PL && make dist`) should generate a META.yml. I'm not sure if it's required since it's a fairly recent addition to the standard distribution tarballs. "What should it profit a man, if he should win a flame war, yet lose his cool?"	[reply] [d/l]
Re: Building a package for distribution: How do the pros do it? by ysth (Canon) on Dec 26, 2006 at 00:09 UTC
In short, use $? only after using some function that sets it; see each particular function for details. $! is more generally applicable to any function taken from the C library that may set an error code.	[reply]
Re: Building a package for distribution: How do the pros do it? by brian_d_foy (Abbot) on Dec 26, 2006 at 22:27 UTC
It looks like you're doing a lot of contortions and gymnastics to keep from developing a better process for getting things done. When you have to do that much work just to keep your pack-rat tendencies, you're really limiting your productivity. The trick is to overcome the things you do and like but that don't really help you. That sets apart the real pro from the amateur. Someone already mentioned MANIFEST.SKIP, in which you can store regular expressions of things Makemaker or Module::Build should ignore. If you move your notes and snippets and whatnot to directories, you can just skip those directories and not worry about adding new files to them. As for your question about `$?` and `$!`, you may want to read perlvar or my "Error Handling" chapter in Mastering Perl. If you want to see how I make distros, you can read "Making New Distributions" from The Perl Journal. Basically, I cook a directory of TT templates. Good luck :) -- brian d foy <brian@stonehenge.com> Subscribe to The Perl Review	[reply] [d/l] [select]