in reply to Re^5: Testing methodology, best practices and a pig in a hut.
in thread Testing methodology, best practices and a pig in a hut.

You're going to have to show some code to prove that.

From Test::Builder v0.78. The bit between big black lines looks an aweful lot like a string eval to me? 

sub cmp_ok {
    my($self, $got, $type, $expect, $name) = @_;

    # Treat overloaded objects as numbers if we're asked to do a
    # numeric comparison.
    my $unoverload = $numeric_cmps{$type} ? '_unoverload_num'
                                          : '_unoverload_str';

    $self->$unoverload(\$got, \$expect);


    my $test;
    {
        local($@,$!,$SIG{__DIE__});  # isolate eval

        my $code = $self->_caller_context;

        # Yes, it has to look like this or 5.4.5 won't see the #line 
        # directive.
        # Don't ask me, man, I just work here.

######################################################################
+##########

        $test = eval "
$code" . "\$got $type \$expect;";

######################################################################
+##########
    }
    local $Level = $Level + 1;
    my $ok = $self->ok($test, $name);

    unless( $ok ) {
        if( $type =~ /^(eq|==)$/ ) {
            $self->_is_diag($got, $type, $expect);
        }
        else {
            $self->_cmp_diag($got, $type, $expect);
        }
    }
    return $ok;
}

[download]

And cmp_ok() is used as the final step of

  1. is_eq
  2. is_num()
  3. _is_diag()
  4. isnt_eq()
  5. isnt_num()

Same file as above. String eval between the thick marks: 

sub _regex_ok {
    my($self, $this, $regex, $cmp, $name) = @_;

    my $ok = 0;
    my $usable_regex = $self->maybe_regex($regex);
    unless (defined $usable_regex) {
        $ok = $self->ok( 0, $name );
        $self->diag("    '$regex' doesn't look much like a regex to me
+.");
        return $ok;
    }

    {
        my $test;
        my $code = $self->_caller_context;

        local($@, $!, $SIG{__DIE__}); # isolate eval

        # Yes, it has to look like this or 5.4.5 won't see the #line 
        # directive.
        # Don't ask me, man, I just work here.

######################################################################
+######
        $test = eval "
$code" . q{$test = $this =~ /$usable_regex/ ? 1 : 0};
######################################################################
+######

        $test = !$test if $cmp eq '!~';

        local $Level = $Level + 1;
        $ok = $self->ok( $test, $name );
    }

    unless( $ok ) {
        $this = defined $this ? "'$this'" : 'undef';
        my $match = $cmp eq '=~' ? "doesn't match" : "matches";

        local $Level = $Level + 1;
        $self->diag(sprintf <<DIAGNOSTIC, $this, $match, $regex);
                  %s
    %13s '%s'
DIAGNOSTIC

    }

    return $ok;
}

[download]

And that function underlies the following api's:

  1. like()
  2. unlike()

Proof enough?

You want rude. This is what I really think of Test::*

Reimplementing a small subset of perl's native functionality--string/numeric comparisons and regex--, using a large, slow, verbose and complex hierarchy of OO code modules, in order to compare scalar variables against constants, when at the final step you are just going to use string eval and let Perl do those comparisons anyway, just so that you can convert the boolean results into a stream of 'ok's and 'not ok's.

And that, just so that you can tie up stdin & stderr in order to produce a set of useless statistic is, in a word, fascile(*).

(*)facile: arrived at without due care or effort; lacking depth;

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

Replies are listed 'Best First'.
Re^7: Testing methodology, best practices and a pig in a hut.
by chromatic (Archbishop) on Feb 28, 2008 at 00:03 UTC

    I suppose "I don't like modules that use string eval" is a fine objection if you want it to be, but I was under the impression that your meditation here was an attempt to argue that blindly applying conventional wisdom as prohibitions without understanding why it's conventional wisdom and why the real world can be more nuanced than it is is a bad thing.

    So color me somewhat confused.

    The bit between big black lines looks an aweful lot like a string eval to me?

    Yep, and that's one way to do it. My Perl 6 reimplementation doesn't use string eval at all, for example. It's unnecessary in Perl 6. I believe my original Perl 5 T::B code dispatched to very small subroutine references. As it turns out, string eval happens to be faster and clearer in Perl 5 than maintaining a comprehensive dispatch table of all of the possible comparison operations. If there's a simpler, faster, and clearer way to do this, I'm sure Schwern would take a patch.

    Similarly, I believe that like() uses string eval so that the code will work on truly ancient versions of Perl 5 that don't have qr//. I don't support versions of Perl older than 5.6.2, but Schwern's willing to do so here. If you have another solution that meets his goals, I'm sure he'd take that patch too.

    However you said "String eval is the underlying basis of what the Test::* modules do." The code you quoted is not proof of that. How can it be? cmp_ok is not the underlying basis of what the Test::* modules do. It's not the underlying basis of Test::More. It's not the underlying basis of Test::Builder. Thus it's not the underlying basis of any Test::* module.

    (What's the underlying basis of T::B? ok() and, to a lesser extent, diag().)

    Reimplementing a small subset of perl's native functionality--string/numeric comparisons and regex--, using a large, slow, verbose and complex hierarchy of OO code modules, in order to compare scalar variables against constants, when at the final step you are just going to use string eval and let Perl do those comparisons anyway, just so that you can convert the boolean results into a stream of 'ok's and 'not ok's.

    My goodness, you're right. It's almost as if we wanted to test Perl code with Perl code. It's almost as if the code in T::B exists to report the locations of failures correctly, handle overloading properly, manage threading appropriately, and provide a uniform interface available from Perl 5.004 or earlier to the latest bleadperl, while allowing hundreds of testing modules to interoperate smoothly in the same process without having to know the internals of even those modules not yet written and without requiring everyone who wants to write tests in Perl to reinvent all of that badly and discover and, hopefully, fix the same weird bugs that everyone who writes testing modules in Perl would have had to fix eventually.

    The horror.

    fascile

    I assume you meant facile, which means "easy", "unconstrained", and "pleasant". Thank you. That's why we wrote those modules.

    (If you meant "superficial", I'd love to see the code you have which solves the same problems. Denying that many people have these problems and have solved them very satisfyingly is not the same thing.)

    As always, you may have the last word in this discussion.

[reply]
[d/l]
[select]

In Section Meditations