The testing survey results are fascinating and, in some ways, discouraging. There is, as always, a bias in online surveys (you have to both know about the survey and have a desire to take it). For example, fully 66% of respondents claimed that their primary code base has a test suite but it's been my experience that the number is far smaller.

I originally posted this to my LiveJournal account, but brian_d_foy offered to put this on Polldaddy under his account. He also expanded a number of areas in the poll and generally made it more useful. Thanks brian.

Also, I would love to expand this survey and post it to my O'Reilly blog to get a better cross-section of programmers, but most of the free survey software out there is limited in the number of responses and/or features. Anyone have any suggestions?

Survey Title: About your test suite...

Page 1

Q.1 Does your primary code base have a test suite?
Answer Count
No 56 (34%)
Yes 111 (66%)
People who answered question: 167 (100%)
People who skipped question: 0 (%)

Page 2

This page was only answered by people who don't have test suites

Q.2 Why don't you have a test suite?
Answer Count
**other** 5 (5%)
Don't have time 26 (26%)
Legacy code 25 (25%)
New Option... 2 (2%)
Don't know how to test 15 (15%)
Too hard to test our code 14 (14%)
Management doesn't approve 12 (12%)
People who answered question: 56 (34%)
People who skipped question: 111 (66%)

Of the five people who answered 'other' to why they don't have a test suite, their reasons were as follows:

  1. Research - exploratory only
  2. Cross-compiling to platform with no testing tools
  3. still learning how to write them
  4. New job where I'm pushing for testing, revision control, etc
  5. mod_perl


Q.3 Do you plan on adding a test suite?
Answer Count
Yes 35 (63%)
No 21 (38%)
People who answered question: 56 (34%)
People who skipped question: 111 (66%)


Q.4 How soon do you plan to add a test suite?
Answer Count
**other** 6 (16%)
Within a year 16 (42%)
Within six months 15 (39%)
This month 1 (3%)
Next Month 0 (0%)
People who answered question: 38 (23%)
People who skipped question: 129 (77%)

Of the six people who answered "other" to how soon they plan to add a test suite:

  1. When hell freezes over or "there are no more features that could be implemented any more"
  2. someday
  3. as soon as we get round tuits
  4. Never
  5. as I have time
  6. Starting now incrementally


Q.5 What are your test suite plans?
People who answered question: 27 (16%)
People who skipped question: 140 (84%)

Of the 27 people who answered this question:

  1. learn how to make a test suite...
  2. Start with basic testing until I understand it better. Then move up to more complex suites.
  3. start from the ground up. Looking at what is a good way to start.
  4. Unfortunately planning and writing test take time and effort. When documentation consists of your ability to read badly written legacy code and management has a incredibly urgent/ninja rush button firmly pressed down on every change request (no matter how small) it is hard to introduce a culture of testing.
  5. None yet.
  6. Start with some of the more easily separated, more sensitive areas of functionality, and build from there.
  7. To use a test suite and learn how to use one.
  8. vague
  9. None
  10. Depends on the organization of the final code. Currently it is a mix of Perl and Java. Once the research phase firms up and we establish a clear direction, we will use a test harness as appropriate for the language.
  11. I'm still learning test based development, I've read a few articles and tutorials, but just started a project that will have test suite from the beginning.
  12. Put tests into my next project
  13. none yet
  14. Refactor code to make it testable. It's a website whose html is generated by print statements and subs return strings of html. So: subs -> data html -> templates
  15. None
  16. If the tools were available, I'd at least start testing my models as soon as we have an initial version of our current programs out.
  17. add tests to existing code as I can; build new code with tests in place.
  18. From-scratch rewrite of large parts is coming up; will have tests. Tests for legacy codebase not considered worthwhile despite regular "oopsies" when rolling out changes.
  19. Haven't given it a ton of thought yet. We have a mixed code base of ASP.NET (VB.NET/C#), PHP, Perl, static HTML, JavaScript, etc and I'd like to either find something that can work between all of these or I need to take the time to get more familiar with various testing frameworks.
  20. write testcases for running code and collect the TAP output for statistics. We need test in the runtime environment, at compiletime there is not much to test (ie: no db, and faking one is no guarantee)
  21. Ridiculously, management pays some junior person to spot-check whatever they feel like checking at the current moment. They think this is an adequate testing policy.
  22. slowly add unit tests as code is touched.
  23. test for every fu**** function
  24. Require tests of all new code. Start with some simple back-fitting of tests. Not sure how to sanely test the large DB queries we do. Not sure how to test the rendering in the front-end.
  25. Start with (a) parts that are used most prevalently (b) parts that are self-contained Write tests for these and gradually work outwards. Adding tests for all new code.
  26. After a code refactoring to allow for more testable chunks, will add in tests.
  27. TAP


Q.6 What would help you get started?
People who answered question: 37 (22%)
People who skipped question: 130 (78%)

Of the 37 people who answered this question:

  1. teach me how to make a test suite
  2. new project
  3. Time
  4. The boss oking it. Several easily accessible tutorials with examples. Good papers on why testing is needed.
  5. Do I use Test::More, Test::Harness, etc? I have lots of stuff to look at but I haven't sat down and figured out where to start, at least what is the current, suggested/approved way to start testing.
  6. Tutorial on options and configuration.
  7. Test::Simple where you can easily write tests in a manner similar to the expect programming language without needing hooks. Most test suites seem aimed at unit tests. Most of the shoddy code I deal with lives outside of modules (usually in web pages).
  8. A formal block of general instruction and guidelines.
  9. Convincing people of that it's worth the time spent.
  10. More time to study.
  11. a framework for testing CMS's af web-applications
  12. Training and more time to devote to testing.
  13. More programmers. :-)
  14. Having more expertise, but I know I can't achieve without actually starting to use them.
  15. A good tutorial.
  16. more time
  17. time from management to work on it
  18. 1. A tutorial on how to test. 2. A "Test Patterns" like "Design Patterns"
  19. More approved time by management
  20. A complete rewrite of our codebase (testing is nearly impossible as is), and management that doesn't consider automated testing to be a complete waste of time.
  21. Someone to come in with a Clue Bat and smack the entire chain of management around until they submit.
  22. Tools for iPhone testing!
  23. - The available book that everyone cites (Perl Tester's Notebook) doesn't work for me as a 'getting started' manual, so having alternate resources would be a great start. Maybe I just don't know where to look. - Time to work on it (this is a corporate issue). - Actual hands-on help from someone who knows what they're doing. I get so stuck sometimes, I dont' even know what to google. - A "testing dojo" at the local PM meeting...maybe I'll suggest that. :)
  24. Having a boss that understands the benefit and cost reductions of implementing automated testing
  25. Hastening the planned feature freeze of the legacy codebase...
  26. Info on popular cheap/free testing frameworks for those languages/frameworks.
  27. documentation on testing strategies. Most talk about ok(4+4,8) but that is not really a testcase IMHO, we also test response times as this is critical to user experience. We could just log this in the db, but I'd like to read about testing strategies for this using TAP (ie: the actual response time is irrelevant, it's either 'good' or 'bad' which can be determined fuzzy wise)
  28. New management. :)
  29. time
  30. time
  31. More time.
  32. Tutorials with non-trivial examples.
  33. How-Tos that show best practices for web site testing - preferably with Catalyst.
  34. A better build system
  35. Understanding testable code.
  36. A raise and a bonus.
  37. A down-to-earth howto.

Page 3

This page and beyond was only answered by people who do have test suites

Q.7 What test framework the test suite is based on?
Answer Count
**other** 25 (17%)
JUnit 11 (8%)
Test::Harness 106 (74%)
XUnit 1 (1%)
ObjcUnit 0 (0%)
OCUnit 0 (0%)
People who answered question: 143 (86%)
People who skipped question: 24 (14%)

Of the 25 people who answered "other" to the "What test framework" question:

  1. Is what I plan to start with
  2. None
  3. Test::Class
  4. Undecided
  5. home-grown c inspired by Test::More
  6. DejaGNU
  7. N/A
  8. N/A
  9. homegrown
  10. home spun
  11. Test::More
  12. schemeunit
  13. N/A
  14. Test::More
  15. Test::More
  16. Test::Class
  17. Test::More
  18. Test.pm
  19. Test.pm
  20. inhouse
  21. Custom. Much simpler and easier than xUnit
  22. Internal
  23. Test::More
  24. NUnit
  25. Mine


Q.8 What is the primary language the test suite is in?
Answer Count
Java 8 (5%)
C / C++ 8 (5%)
**other** 8 (5%)
Python 3 (2%)
Perl 124 (82%)
Ruby 0 (0%)
People who answered question: 151 (90%)
People who skipped question: 16 (10%)

Of the 8 people who answered "other" to the "primary language" question:

  1. Would be perl, but you have no skip ahead option in the poll
  2. N/A
  3. Undecided
  4. scheme
  5. N/A
  6. JavaScript
  7. C#
  8. Scheme

Page 4

Q.9 How many tests are in the largest test suite you work with?
Answer Count
Fewer than 50 tests 32 (22%)
50 to 100 tests 19 (13%)
101 to 500 tests 35 (24%)
501 to 2,000 tests 25 (17%)
2,001 to 10,000 tests 16 (11%)
More than 10,000 tests 16 (11%)
People who answered question: 143 (86%)
People who skipped question: 24 (14%)


Q.10 How long does your test suite take to run?
Answer Count
Fewer than 30 seconds 31 (22%)
30 seconds to 1 minute 23 (17%)
1 minute to 5 minutes 36 (26%)
5 minutes to 10 minutes 9 (7%)
10 minutes to 20 minutes 13 (9%)
20 minutes or more 26 (19%)
People who answered question: 138 (83%)
People who skipped question: 29 (17%)


Q.11 What percentage of your code is covered (rounding down)?
Answer Count
Don't know, but I do coverage testing 6 (4%)
Don't do coverage testing 47 (34%)
**other** 3 (2%)
Less than 50% 21 (15%)
50% to 74% 18 (13%)
75% to 84% 16 (11%)
85-94% 12 (9%)
95-99% 15 (11%)
100% 2 (1%)
People who answered question: 140 (84%)
People who skipped question: 27 (16%)

Of the three people who answered "other":

  1. What tests?
  2. less than 1%
  3. A tiny fraction, so far


Q.12 Do all tests pass?
Answer Count
No 28 (20%)
Yes 110 (80%)
People who answered question: 138 (83%)
People who skipped question: 29 (17%)

Some of the results I found particularly interesting are as follows (I present hard numbers, but it's worth remembering that they are all in relation to those who took the survey):

Cheers,
Ovid

Replies are listed 'Best First'.
Re: Test Survey Results
by perrin (Chancellor) on Aug 03, 2008 at 00:59 UTC
    I'm kind of bummed to see "mod_perl" listed as a reason for not having tests. The mod_perl team put a lot of work into Apache::Test so that people would write tests for their mod_perl apps.
[reply]
      It's really whoever gave that answer who should be "bummed". They're clearly giving an invalid excuse for not doing testing. There are thousands of reasons you can give for not doing testing, but only a few of them are genuinely valid.
[reply]
        Perhaps they don't know about Apache::Test. That makes it a *reason*, not the "invalid excuse" that you so blithely dismiss. For all I know, they might have other *reasons* that mod_perl makes testing hard too.
[reply]
Re: Test Survey Results
by dHarry (Abbot) on Aug 03, 2008 at 09:54 UTC

    First of all well done!

    I like the topic and I know from experience how laborious gathering statistics can be.

    I would also like to use the opportunity to elaborate a little on the sample size (the 167 participants) as well.

    Sample size

    When undertaking any sample survey (a collection of information from only part of a population), you may experience what is known in statistics as a “sampling error”.

    The sampling error can be defined as the difference between the estimate derived from a sample survey and the “true” value that would result if a 100% sample survey (the whole population) were taken under the same conditions.

    The sample of 167 people seems (very) small to me. How many programmers are out there? What sample size is necessary to be able to reliably say something about the population?

    I welcome your idea to expand the test and put it on O’Reilly.

[reply]

      The spread from sample size is likely dwarfed by selection bias in this case.*

      "For example, fully 66% of respondents claimed that their primary code base has a test suite but it's been my experience that the number is far smaller." is where the selection bias shows. I agree that the real number is much smaller, and what we are seeing is that people who like to talk about test suites are likely to have test suites.

      * - The simple way to estimate your standard deviation from a sample is the bootstrap method: you assume the population is like your sample and just use the deviation of the sample. The deviation of a rate (means, percentages, etc.) goes as the deviation of one draw divided by root the number of draws. So, in this case, about 3.8%. Even if you had n=1000, your spread would be ~1.5% or so. As a poll taker, your greatest challenge is much more often how your select your sample than it is the sample size.

[reply]

      You should also point out the sample distribution as well. I have known about 20-30 perl programmers that I have worked with and none of them have ever done testing. Nor have any of them ever visited perlmonks.org or use.perl.org or any other organized mailing list. CPAN is considered some type of spam site(maybe I am exaggerating a little).

      The point is, that the people who would see and take the Poll as people who are more active in the community therefore more likely to have better habits than someone who 'already knows' how to do programing and therefore does not consult the web for advice. This distribution will royally screw up any potential estimate that you are trying to get from your Poll.

[reply]
Re: Test Survey Results
by Anonymous Monk on Aug 02, 2008 at 19:29 UTC
    You have a bias that is showing. I laughed when I read "More than 10,000 tests" but nothing beyond "20 minutes or more". I was just running one test suite (of many) that has more than 10,000 tests (a few 100,000) which runs more than 5 hours. At my last job our full UT test suites ran several hours as well.
[reply]

      I can assure you that many people who have extensive experience with testing will detect far more issues than just that (for example, test counts in xUnit versus TAP systems). I won't deny that outliers such as what you discuss are important, but I'm more interested in the common case right now. Outliers are generally handled differently because they are different.

      Cheers,
      Ovid

[reply]
      :; time make test
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_h
+arness(0, 'blib/lib', 'blib/arch')" t/*.t
t/01-conversion_0_10000................ok           
t/02-conversion_10001_999999...........ok       
t/03-conversion_1000000_and_up.........ok     
t/04-special-cases.....................ok   
t/71-version_numbers_are_the_same......ok   
t/pod-coverage.........................ok   
t/pod..................................ok   
All tests successful.
Files=7, Tests=10218,  7 wallclock secs ( 0.94 usr  0.15 sys +  6.70 c
+usr  0.14 csys =  7.93 CPU)
Result: PASS

real    0m7.180s
user    0m7.726s
sys    0m0.324s

      [download]

      There you go, over 10k tests in just 7 seconds.

      Granted, that is not a full system, just a module. However, my answers remain: my largest test suite has over 10000 tests, and it runs in just a few seconds.

      I also recall creating a system with ~400 tests that would run in just no time. Then again, the most enforced requisite of that platform was that it had to process over 1000 requests per second.

      And it did.

[reply]
[d/l]
        Well... 
        ambs@rachmaninoff ProjectoDicionario$ time make test
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0)" t/*.t
t/empty_lines....ok                                                          
t/last_line......ok                                                          
t/test_terms.....ok                                                          
All tests successful.
Files=3, Tests=67546, 16 wallclock secs (13.43 cusr +  1.40 csys = 14.83 CPU)

real	0m15.786s
user	0m18.537s
sys	0m1.806s

        This is not to show tests run quickly, as the tests are quite simple. What I would like to say is that these tests in particular are not testing code, but testing data quality (a PT dictionary formatting).

        Why is this helpful? Because when I too the time to write the tests, I found out that I had more than 6000 entries with formatting errors.

        Alberto Simões

[reply]
        Obviously one can have a ton of trivial tests that run in very little time. That doesn't demonstrate a lack of bias. The bias was not contemplating the possibility that an extensive test suite could take much more than 20 minutes to run. In my experience, extensive test suites on a company-wide scale usually take quite a bit more than 20 minutes to run, so I don't consider such to be an 'outlier' as characterized by Ovid to dismiss the criticism. But I'd not have set the upper limit on number of tests at 10,000 either. I'm sure we each have over 100,000 tests at my current and previous employer.
[reply]
Re: Test Survey Results
by EvanCarroll (Chaplain) on Aug 14, 2008 at 19:15 UTC
    There is one other type of bias I believe to be in your report. There is no easy way to say "no". I don't have a test case for my "primary code" as you ask... And when I said "no" you asked me questions presuming I had... If I don't fill out these questions, does my initial "No" still count against your 66% testing rate... For instance, I stopped on page 3... I would say depending on your answer this whole survey could be so bunk as to be totally useless... And I agree, the real question is "where is the bias?"... While, I think I have identified it, I also think posting these results to the world would just solicit laughter by people more capable of conducting a survey. There is obviously something profoundly wrong here...

    A more accurate representation would be to assume the code on CPAN is representative of people's production code... Now test the CPAN dists for tests... Here the bias is simple: the average CPAN programmer is more capable than the average programmer, by virtue of being a programmer AND knowing how to navigate CPAN... Which in my eyes has always been an issue. The learning curve to get something on CPAN, or give back your changes to CPAN is always more bureaucratic and difficult than making the simply making the change and keeping it to yourself. I would venture out and say no more than 20% of all Perl code under a build system (M:I, M:B, EU:MM) is under a test suite, and less than 20% of all Perl code is under a build system.


    Evan Carroll
    I hack for the ladies.
    www.EvanCarroll.com
[reply]

Back to Meditations