"deep" unit testing

LanX has asked for the wisdom of the Perl Monks concerning the following question:

Warning: lots of magic ahead!

My colleague came up with a question regarding a very special case of unit testing and I'm not sure how to answer it.

there are loads of legacy routines which produce complicated data-structure and write them to file.
he doesn't want to test the very complicated output but the intermediate datastructures without changing the code much

My response was the clean approach is to refactor the code into additional subs called from the "printers" returning the data structures and test those nested "units".

Obviously the code would be easier to maintain and reusable.

He declined it, cause he is very critical about performance and doesn't want the overhead of extra sub calls.*

So I suggested a "dirtier" approach to use a constant TEST to flag that tests are running and to safe the desired code into package variables named "$PKG::TEST_something" (or probably $PKG::TST::Something)

The advantage is that every code conditioned by ... if TEST won't show up in production code.

Here a condensed demo in a single file, the output of B::Deparse shows that the TEST code disappears. (In a real test.t file the "PKG::TEST" flag has to be set before requiring the module PKG to be effective)

Question Is there a better way to do this?

#  set TEST flag before compiling tested module
use constant 'PKG::TEST' => 1; 

# ###  here normally use PKG
{
    package PKG;

    use strict;
    use warnings;
    use Data::Dump qw/pp dd/;
    use v5.16;





    sub create_output {
        my $count =shift;
    
        my $string;
        my $fh;

        my %hash;

        # create (mildly) complex demo HoA
        $hash{$_} = [reverse 1.. $count--]
            for "a".."f";

        # memorize datastructure when testing
        $PKG::TEST_result = \%hash if TEST;

        my $target = "c:/tmp/output.txt";

        #  redirect filehande to string when testing
        $target = \$PKG::TEST_content if TEST;
    
        open $fh, '>', $target;


        #  Output to "file"
        print $fh pp \%hash;
    }
}


# ### my tests
package main;

use B::Deparse;
use Data::Dump qw/pp dd/;


# prove that TEST code disappears unless TEST is true 
print B::Deparse->new->coderef2text(\&PKG::create_output);

PKG::create_output(5);

if ( PKG::TEST ) {
    use Test::More;

    my $expected ={
        a => [5, 4, 3, 2, 1],
        b => [4, 3, 2, 1],
        c => [3, 2, 1],
        d => [2, 1],
        e => [1],
        f => [],
    };
    
    is_deeply(
        $PKG::TEST_result,
        $expected,
        "internal result"
    );

    my $expected_str = pp $expected;
    
    is(
        $PKG::TEST_content,
        $expected_str,
        "file output"
    );

    done_testing();
}
[download]

The problems I see:

still litering production code with TEST statements
when testing multiple subs which call each other they may confuse the results
there doesn't seem to be a way to check the subs name at compiletime
__SUB__ gives me only the subref which isn't known at test start.
I can only use __SUB__ as a hash key to store different results...

Cheers Rolf
_{(addicted to the Perl Programming Language :)

Wikisyntax for the Monastery
FootballPerl is like chess, only without the dice}

*) a case were inlining would be nice, more about that in a future post

update

added comments to code

Comment on "deep" unit testing Select or Download Code

Replies are listed 'Best First'.
Re: "deep" unit testing by stevieb (Canon) on Mar 27, 2019 at 20:43 UTC
I'm literally headed out the door to go out and tune/prep my boat motors so I can drop my boats out in the water this weekend, so I don't have time to review thoroughly or respond properly until later. That said, in response to this: "...cause he is very critical about performance and doesn't want the overhead of extra sub calls." ...I call that ridiculous horse shit (or laziness, take your pick, but I don't think the "Lazy" that Perl implies relates to such a situation).	[reply]
Re^2: "deep" unit testing by LanX (Saint) on Mar 27, 2019 at 20:46 UTC
Well sub calls in Perl are expensive. * I don't know the special case though. and since it's legacy code this approach might bring better results than trying to refactor hundreds of time critical night jobs. Cheers Rolf _{(addicted to the Perl Programming Language :) Wikisyntax for the Monastery FootballPerl is like chess, only without the dice} update *) see also RFC: Inline::Blocks or inline as a keyword?	[reply]
Re^3: "deep" unit testing by stevieb (Canon) on Mar 27, 2019 at 20:52 UTC
Can you elaborate a bit on what we're talking about in regards to timing? 5ms, 100ms etc? Can you also explain in a bit more detail the nature of what is being tested? No need to go to extremes here, just enough so we have a better idea of the why regarding such time-sensitive (and non-additive-to-codebase) testing. Is mocking out pieces of specific tests an option? Do you store data externally so that you can test against it so timing isn't so critical?	[reply]
Re^4: "deep" unit testing by LanX (Saint) on Mar 27, 2019 at 20:58 UTC
Re^5: "deep" unit testing by stevieb (Canon) on Mar 28, 2019 at 02:21 UTC
Some notes below your chosen depth have not been shown here
Re: "deep" unit testing by haukex (Archbishop) on Mar 28, 2019 at 20:33 UTC
A couple of thoughts... I would also call into question the statement that adding `sub` calls would slow things down, on its own that smells like it could be premature optimization and/or cargo culting. How often does this code get called? Has the performance difference of inlining the code been tested? And even if it's slower, is it significant? I.e. if a script runs 1.5 minutes instead of 1 minute, does that really matter if the script only gets run once a day? Although using a constant to optionally add in code is probably a somewhat decent solution, some might argue that by adding test code that is optionally compiled in, one is actually altering the code that is being run, i.e. you've got different code running during testing and a normal run. Although I'm having trouble finding the reference at the moment, I know that there was at least one thread here not all too long ago that mentioned at least one good CPAN module that made testing even complex data structures possible (it allowed defining a kind of "schema" for Perl data structures, IIRC). Asking the other way around, what is the concrete argument against treating the `sub` as a single unit and testing the entire output data structure? What do these output structures contain that would make automated testing so difficult? (For example, if it's timestamped data that's hard to nail down, then could you add mocks such that the time is always the same during the tests?)	[reply] [d/l] [select]
Re^2: "deep" unit testing by LanX (Saint) on Mar 28, 2019 at 22:28 UTC
There are always cases where you can't avoid whitebox testing. I'm limiting the inserted test code to the absolute minimum, any print, dump or log would have the same damage. ( But now I understand the point of Test::Inline better to generate test code out of embedded POD) I wished I could go into a deep analysis of the code base and help sketching some architectural advice which is followed by all programmers here. Unfortunately that's not how it works and not what I was hired for. It would involve weeks of discussions and arguments with management and colleagues, of whom many are only self-trained part-time programmers. At the same time I have to deliver the projects I was commissioned with. We are already glad to have introduced testing and real version control for the time being. Cheers Rolf _{(addicted to the Perl Programming Language :) Wikisyntax for the Monastery FootballPerl is like chess, only without the dice}	[reply]