Is there a test harness that doesn't suck? Any thoughts on a custom built test harness that is badly written in Java? Any test harness gripes? How difficult is it for you to stop coding so that your test meets the harness as seemlessly as the harness will allow?

Clarification on that last one: you have to not code your own test harness since you already have one provided for you. Sometimes I just start coding harness functionality without thinking about it, and I have to stop myself.

Any gripes on limitations of your test harness? Dopey scenario sytax? Ridiculous work arounds you've had to do?

I'm integrating my stuff into our test harness now, and I'm not happy with the harness we're using. {groan}

Update:

I'm not talking about the Perl module Test::Harness. I amtalking about the concept of test harnesses, in general. You know, like Silk Test or LoadRunner, but more specifically, in-house custom-built test harnesses with which you are forced to work. Also, I'm not looking for help, that is why I posted in Meditations, not Seeking Wisdom.

I can see how the confusion started that I might have been talking about CPAN's Test::Harness, but I wasn't, and I've never used it. Anyhoo, my original questions still stand. Any meditations on those?

  • Comment on Test Harnesses (in general) -- Meditation of Dissatisfaction

Replies are listed 'Best First'.
Re: Test Harnesses Meditation of Dissatisfaction
by Corion (Patriarch) on Mar 08, 2006 at 08:37 UTC

    I'm not exactly sure what you're getting at, but so far, I've been pretty much content with Test::Harness. If I had to test anything other than Perl, I would go either with the native test environment or, if that one doesn't work out, with the Test Anything Protocol, which specifies the output just like Test::Harness, so that Test::TAP::Model can generate pretty reports from it.

[reply]
      Sorry for the confusion. See my update to original post. There is no way on God's green earth that this dept is gonna be using CPAN's Test::Harness. Either the current Java one must get better or...well, or nothing.
[reply]
Re: Test Harnesses Meditation of Dissatisfaction
by adrianh (Chancellor) on Mar 08, 2006 at 11:08 UTC
    Is there one that doesn't suck? Any thoughts on a custom built test harness that is written in Java (barf)? Test harness gripes? How difficult is it for you to stop coding so that your test meets the harness seemlessly as the harness will allow? Clarification on that last one: you have to not code your own test harness since you already have one provided for you. Any gripes on limitations of your test harness? Dopey scenario sytax? Ridiculous work arounds you've had to do?

    I work quite happily with Test::Harness and prove most of the time. I've got a few custom Test::Harness::Straps based runners for particular issues but generally I've found the Perl test harnesses to do what I need.

    What problems are you having?

[reply]
Re: Test Harnesses Meditation of Dissatisfaction
by jffry (Hermit) on Mar 08, 2006 at 13:19 UTC
    I'll go ahead and reply to myself, so folks get an idea of what I mean by this meditation.

    I happen to know of a department in a company that uses a test harness written in Java. Except for heavy stress tests, can you guess which gets more CPU time: the test or the harness? Do I even have to tell you? When I asked why this dept was using a CPU heavy/Java test harness rather than one written (and written well, of course) in C, I was told (under the table) that long ago, it was a political decision! It would have hurt more people's feelings to choose the C one over the Java one!

    OMG!

[reply]
      There's no reason that an efficient Java test harness couldn't be written. It sounds like the one one you have is just bloated and over-designed.
[reply]
        Oh, you bet! In another dept in this same company, there is a test harness (coded in-house, of course) that is Java and isn't nearly so bad. In fact, it works rather well, so I'm told. However, this dept couldn't switch now without major upheavals, due to all the additional tools built to support and leverage it. Ug!

        Luckily, this dept. now has a good Java programmer on staff, so it is slowly getting better. The original coder wrote the test harness because he "wanted to learn Java." Oh, boy!

[reply]
      When I asked why this dept was using a CPU heavy/Java test harness rather than one written (and written well, ot course) in C, I was told (under the table) that long ago, it was a political decision! It would have hurt more people's feelings to choose the C one over the Java one!

      Well if the code being tested is Java I'd go the Java route too. Keeping the testing framework and the code being tested in the same language usually makes things much easier.

      If I were you I'd just start things moving to one of the decent Java frameworks. TestNG is very nice, and the latest JUnit is pretty good too.

[reply]
        I'll go ahead and suggest that our current test harness maintainer take a look at these Java test harnesses. Maybe he can get something useful from them.

        Again, I can't make major architectural decisions on the test harness. About the most I can do is request features if I can justify them. Although, in the next few years I might be able to make a case for those other Open Source harnesses you suggested.

        As for writing the tests in Java just because the harness is writen in Java, well, it doesn't really matter. The harness-to-test interface is language independent. It has it's own syntax, and coding in Java gains no added benefit.

        But thanks again for suggesting JUnit & TestNG. That may very well turn the corner for our dept in the years to come. I mean it. Sincerely.

[reply]

Back to Meditations