in reply to CPAN module unit test issues: OS line endings

How did you write such fragile or broken code that it breaks in the face of different (but native) line endings despite Perl pretty much taking care of that for you automatically? :)

I could see having a problem on an old Mac if you included test data in files having non-old-Mac line endings (but perhaps Perls on old Macs handle that case better than I would expect based on the line-ending logic that I am aware of as I have not looked at the old-Mac-specific parts of Perl's source code). And I could see having minor problems when including data files having Windows line endings and then using them on Unix (if you don't always strip trailing whitespace from text lines, as is wise).

With your (subsequent) mention of Devel::Examine::Subs and finding test failures on Windows (for 10 versions back), the specific failure cases appeared to me to be types of failures that would be rather far downstream from the direct impact of different line endings, so I didn't continue spending time trying to figure out how the failures happen. Perhaps you could just describe that to us?

- tye        

  • Comment on Re: CPAN module unit test issues: OS line endings (how?)

Replies are listed 'Best First'.
Re^2: CPAN module unit test issues: OS line endings (how?)
by stevieb (Canon) on Sep 18, 2015 at 01:37 UTC

    The problem stemmed from the fact that Tie::File (which I use in this module) switches its record separator automatically when on either nix or Windows. So when my test data files were downloaded from CPAN on Windows, they still had Unix line endings and weren't being read in correctly.

    In the end, before I open a file with Tie::File, I check to see what type of line endings it has, then I set Tie::File's recsep parameter to either \n or \r\n depending on the situation.

    Note that I hardly ever use Windows, so didn't think to test it on that platform, and of course we haven't been getting Testers reports because of the issues.

    I'm going to look at Tie::File tomorrow to check whether I'm missing something, or whether automatic record separation can be incorporated within it (just \n, \r\n).

[reply]
[d/l]
[select]

      Why are you using Tie::File? I can't recommend that module as it is a fine example of getting a slight bit of superficial simplicity at the expense of way too much hidden complexity. Such things too often end up biting you before you are done (as happened here).

      I only skimmed a bit of the code. The only actual using of a tied array that I noticed was: 

      @{$subs{$file}{TIE_file}} = @TIE_file;

      [download]

      Which seems to provide absolutely zero benefit from the use of that module. That does nothing more than what a simple open and then @{...} = <$fh>; would do (except it is less efficient and leverages way more hidden complexity which leads to fragile surprises like not dealing well with line endings).

      That code also slurps the entire file contents into memory. This limits the size of problems that can be effectively handled by your module.

      - tye        

[reply]
[d/l]
[select]
        (except it is less efficient and leverages way more hidden complexity which leads to fragile surprises
        ++ x 1000

        Cpanitis in a nutshell.

[reply]

        I started off wanting to edit files in place by array element, and found Tie::File was nice and easy. As I expanded with the project, I guess I just went overboard and used it everywhere.

        Sometimes we just need someone more experienced to say "whoa, wait a minute, you're being ridiculous". :)

[reply]
[d/l]
Re^2: CPAN module unit test issues: OS line endings (how?)
by syphilis (Archbishop) on Sep 18, 2015 at 03:11 UTC
    How did you write such fragile or broken code that it breaks in the face of different (but native) line endings despite Perl pretty much taking care of that for you automatically?

    It could well be that tar is the fly in the ointment.
    When downloading from CPAN one is generally grabbing tarballs, and the text files in those tarballs most commonly have nix line endings.
    Most Windows tar utilities can, I think, be configured to convert the line endings to Windows format but I've been bitten by configuring tar that way. (On rare occasions it would consider a binary file to be text.)

    As a consequence of that, I for one, always untar CPAN tarballs "as is" on Windows - which means that the unpacked text files most commonly contain nix endings.
    And I doubt that I'm the only person doing that.

    This makes it difficult for a CPAN author to know whether the unpacked distro on a Windows machine will have nix or windows line endings - and I think the best general advice is to construct things such that there's no need to know this.

    Cheers,
    Rob
[reply]
[d/l]
[select]
      Most Windows tar utilities can, I think, be configured to convert the line endings to Windows format but I've been bitten by configuring tar that way.

      Getting either Unix or Windows line endings in text files when on Windows should not present a problem (to mundane Perl code).

      - tye        

[reply]
        Getting either Unix or Windows line endings in text files when on Windows should not present a problem (to mundane Perl code).

        I'm not suggesting otherwise.

        Cheers,
        Rob
[reply]

In Section Seekers of Perl Wisdom