What should be captured in automated test reports?

As I've been developing CPAN::Reporter, I've been thinking about what information I wished I had when examining failure reports on CPAN Testers. Test::Reporter already includes the results of "perl -V". As part of CPAN::Reporter, I include a list of prerequisites specified and the module versions loaded (and I believe that CPANPLUS does this as well).

Now I'm considering what else might be useful. From an initial small list I had, I've brainstormed a longer list of additional things that might be useful to include. For example:

Toolchain module versions

It seemed like it might be useful to know what versions of the module toolchain are in use. These aren't listed in prerequisites and could be a potential source of test problems (particularly false negatives).

CPAN
CPANPLUS -- though not really relevant for CPAN::Reporter
ExtUtils::MakeMaker
Module::Build, along with ExtUtils::CBuilder and ExtUtils::ParseXS

Environment variables

Many test programs change behavior based on environment variables. In addition, it seemed to me that some environment variables might answer other questions about the operating environment at the time of the failure.

qr/PERL/ -- anything specific to Perl and kind enough to flag it in the name of the variable
PATH
SHELL
TERM
AUTOMATED_TESTING -- used by many authors to toggle tests for non-interactive situations
AUTHOR_TESTING -- I use this for pod/coverage tests, though I probably should have called it PERL_AUTHOR_TESTING
INCLUDE -- win32 gcc path to header files
LIB -- win32 gcc path to library files
LD_LIBRARY_PATH -- GNU/linux path to library files

Other stuff

This is more of a catch-all, but it seemed like many of these might be useful to know when investigating a test failure report.

perl executable name -- $^X -- helpful for detecting spaces/non-standard locations
UID/EUID and GID/EGID -- $<, $>, $(, $) -- running as root or as a normal user?
umask -- affects files created during tests
Win32::GetOSVersion() -- only for win32, of course, but would give more detail than just "MSWin32" from $^O (e.g. Win98? Vista?)

Program versions

This is a much more speculative idea, as there is no standard way to ask for a version for an external program. It might or might not really matter except in very unusual XS cases.

compiler
$Config{make}

Feedback needed

I'm interested in what people think of these lists. Which of these seem useful? Which are overkill? Are there things missing that you'd really like to have in a test report. Which would you prioritize?

-xdg

Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.

Comment on What should be captured in automated test reports? Select or Download Code

Replies are listed 'Best First'.
Re: What should be captured in automated test reports? by demerphq (Chancellor) on Sep 24, 2006 at 19:30 UTC
Minor comments: Instead of or as well as Win32::GetOSVersion I'd say provide the result of `ver`. Neither LIB nor INCLUDE are gcc specific on Win32. They are standard for VC as well. Instead of umask, it might be interesting on Win32 to see the results of stuff like cacls. Knowing cwd() would be useful. As would knowing the full paths to things like (n)make, and other tools. Sometimes failure could be occuring because the tools have spaces in their names. On win32 knowing if the user is an administrator/power user or not would be useful. Locale settings might be nice. For instance code might be failing because the locale has strange settings for the thousands seperator or the decimal point. If it was possible knowing the machine localtime as well as a reference time would be good. Although i could imagine this would be a difficult one. As for prioritization, I'd say that the more info provided the better so prioritize the easiest to add, and defer the harder until the rest are done. --- $world=~s/war/peace/g	[reply]
Re: What should be captured in automated test reports? by Corion (Patriarch) on Sep 24, 2006 at 19:38 UTC
When logging, there is no such thing as overkill. That's why god gave us `grep`. The full, verbose output of the build log and of the tests is indispensable. Possibly, as `dmake` has just seen a new version, the version number of `$Config{make}` could be of interest too, especially as your focus is on non-MS Win32 Perl distributions. I thinkt that MSVC also uses $ENV{INCLUDE} and $ENV{LIB} to indicate the include and library directories. One thing of possible other interest could be the number of CPUs the machine has, just to track down potential race condition problems causing tests to fail - on Win32, this would be via $ENV{NUMBER_OF_PROCESSORS}, at least on Windows 2000 onwards.	[reply] [d/l] [select]