make dist (ExtUtils::MakeMaker) on Windows stores text files with DOS line endings in .tar.gz archive

bart has asked for the wisdom of the Perl Monks concerning the following question:

When preparing an upload to CPAN, I noticed that

make dist
[download]

doesn't convert the text files to Unix format before creating the .tar.gz archive file. I'm asking here: does this matter?

Perl used to care about line endings in the source files, not even so long ago (= I'm sure there are still perl versions in use to which line endings matter). It no longer does, except at the end of a shebang line in a script, perhaps.

Still, I don't think it's how it's supposed to be, for archives on CPAN.

Hunting around in the guts of ExtUtils::MakeMaker, I read:

make tardist
First does a distdir. Then a command $(PREOP) which defaults to a null command, followed by $(TOUNIX), which defaults to a null command under UNIX, and will convert files in distribution directory to UNIX format otherwise. Next it runs tar on that directory into a tarfile and deletes the directory. Finishes with a command $(POSTOP) which defaults to a null command.

make dist
Defaults to $(DIST_DEFAULT) which in turn defaults to tardist.

n.b. Do note the typo: the docs say $(TOUNIX), while actually it should be $(TO_UNIX). I reported that documentation bug earlier today.

Apart from OS/2, none of the platforms seem to define a setting for $(TO_UNIX), and OS/2 does it in a most bizarre way (you have to "view source to see this):

sub init_dist {
    my($self) = @_;

    $self->{TO_UNIX} ||= <<'MAKE_TEXT';
$(NOECHO) $(TEST_F) tmp.zip && $(RM_F) tmp.zip; $(ZIP) -ll -mr tmp.zip
+ $(DISTVNAME) && unzip -o tmp.zip && $(RM_F) tmp.zip
MAKE_TEXT

    $self->SUPER::init_dist;
}
[download]

Yowza: it seems to be using the ZIP command line tool, to decide which files are textfiles, and convert the line endings for those!

My questions: Are the tarred up archived indeed Windows text files? Or does "something" magically convert the files to Unix? For example, PAUSE seems to be doing some additional processing before adding the archive to CPAN. Does it take care of these line endings too? If they are, and it doesn't: does it matter? If it matters, how do we patch it? I feel no inclination to copy the code used for OS/2.

Thank you in advance for any suggestions.

Comment on make dist (ExtUtils::MakeMaker) on Windows stores text files with DOS line endings in .tar.gz archive Select or Download Code

Replies are listed 'Best First'.
Re: make dist (ExtUtils::MakeMaker) on Windows stores text files with DOS line endings in .tar.gz archive by syphilis (Archbishop) on Nov 20, 2006 at 00:09 UTC
I'm asking here: does this matter? I don't think it matters. As I understand it, PAUSE just accepts the tar.gz as is. It's up to the tar utility of the user who downloads the tar.gz to handle the text files appropriately when unpacking the archive. Cheers, Rob	[reply]
Re^2: make dist (ExtUtils::MakeMaker) on Windows stores text files with DOS line endings in .tar.gz archive by randyk (Parson) on Nov 20, 2006 at 06:30 UTC
I think that's right that PAUSE doesn't care, and also, it's up to the user's system to handle the line endings appropriately. However, there may be some circumstances where unix-style line endings are expected in a distribution - for example, some time ago search.cpan.org had to patch some tools that generate the html docs if Windows-style line endings were present, and there was a recent bug in ExtUtils-Manifest also related to such line endings. It's probably best, if possible, to run make dist to create a CPAN distribution on a Unix machine.	[reply]
Re^3: make dist (ExtUtils::MakeMaker) on Windows stores text files with DOS line endings in .tar.gz archive by Corion (Patriarch) on Nov 20, 2006 at 07:57 UTC
Running `make dist` on a Unix machine doesn't help anything if the text files have Windows-style (or Mac-style) line endings. The only thing that would help would be to convert all files to Unix-line endings before running `make dist`, which can be done on a Windows or Unix machine. I use a test file by brian_d_foy that checks the line endings of all files in my distribution to ensure that all are consistent.	[reply] [d/l] [select]
Re^2: make dist (ExtUtils::MakeMaker) on Windows stores text files with DOS line endings in .tar.gz archive by bart (Canon) on Nov 20, 2006 at 11:25 UTC
It's up to the tar utility of the user who downloads the tar.gz to handle the text files appropriately when unpacking the archive. That sounds weird. I had half and half expected that the tar utility would handle it, when creating the .tar(.gz) file. But apparently, it doesn't, as the CRs are present in the archive. But, it still feels like a more reasonable expectation to me, because you (and your make) knows what platform you're on when building the archive, so converting it to a common format is easy. The client has less info, he doesn't actually know what platform it originates from, so converting it from whatever into his own native format involves more guesswork.	[reply]
Re^3: make dist (ExtUtils::MakeMaker) on Windows stores text files with DOS line endings in .tar.gz archive by syphilis (Archbishop) on Nov 20, 2006 at 11:59 UTC
I had half and half expected that the tar utility would do it, when creating the .tar(.gz) file I'm thinking that when the tar utility creates the archive it should do nothing other than pack the files it has been given ... and that gzip should do nothing other than gzip the archive. (Why should the tar utility convert to nix endings when, for all it knows, the tarball is never going to be unpacked onto a nix machine anyway ?)The clever bit needs to happen when the archive is unpacked - at which point the tar utility looks at the line endings and adjusts them as needed (iff needed). Personally, on Win32, I make sure that the tar utility leaves the line endings as they are - that way I don't have to worry about the tar utility amending files that it shouldn't. (I have been bitten by this.) Worst case scenario (for me) is that I then have to view the text files in Wordpad (rather than Notepad) to have them dispayed in a sane fashion. Cheers, Rob	[reply]
Re^2: make dist (ExtUtils::MakeMaker) on Windows stores text files with DOS line endings in .tar.gz archive by Argel (Prior) on Nov 21, 2006 at 18:30 UTC
Tar is a backup and archival utility (name comes from "Tape Archive") and as such should not be making any modicifcations to the file while adding it to the archive or while extracting it. Modifying the files goes directly against the intended use for the command. There are other tools for doing text file conversion such as dos2unix and unix2dos. Update: Fixed typos	[reply]