Test code that makes filesystem modifications?

I posted this to the modules mailing list, but the signal/noise ratio on that lists seems to have fallen through the floor, so it doesn't surprise me that I got now responses.

I'm attempting to write some test code for the Win32::Security modules, and I'm beset by a conundrum.

To really test something that is designed to manipulate the security of files on the filesystem, one has to really do that. Otherwise, it's just smoke and mirrors and you don't know if the code really works.

I've flipped back and forth over this issue. Should I knowingly put code in a test suite that intentionally makes changes to the state of a system? I've started playing with such a test suite, and I've decided that if I decide to do this, the best approach is one like:

Create the directory "$ENV{TEMP}\\Win32-Security_TestDir_$$" and do all my testing in there
Wrap all of the test code in an eval {}; and have it do its work in and on that directory
After the eval, do a system("rd /s /q \"$tempdir\""); (where $testdir is the directory created above)

Other requirements for the code are that it be able to identify the user running the test (using $ENV{USERDOMAIN} and $ENV{USERNAME}) and that it be able to find cacls.exe (which wasn't standard with the OS until Win2K, IIRC). Since I originally posted the message, I've started writing test code using the above approach. I'm finding oodles of corner-of-the-envelope bugs (and fixing them) - black-box reinvention of Microsoft's internal ACE propagation algorithms is proving to be non-trivial. I'm glad I've started doing this, the only question is whether I should release the tests or somehow provide them in a disabled form for users who really want to test. It would be nice to get feedback from CPAN Testers, though.

So, should I risk this? Is this a reasonable solution? How and where should I put appropriate warnings?

Thoughts?

In another vein, I'd like to write a test suite for some scripts I distribute with the modules (PermDump.pl, PermChg.pl, and PermFix.pl). I've placed these scripts in the lib/Win32/Security directory so that they will get packaged up and distributed by PPM packages (I hope), since they are very useful tools based upon this module.

What is the best way to locate those modules so that I can test them from a .t file? Scanning @INC for the files? How do I know what version of perl.exe to call (remember I'm running under Win32, and I'd like this module to work on any build since 5.005)?

Finally, I need another account to assign permissions to for the testing. Do you think I'm safe using Guest (since it's a BUILTIN and present on every machine)? What about localization issues? I think these accounts get different names under different localized versions of Win32. Aiiii!!!!

--Toby Ovod-Everett

Comment on Test code that makes filesystem modifications?

Replies are listed 'Best First'.
Re: Test code that makes filesystem modifications? by Abigail-II (Bishop) on Mar 01, 2004 at 21:20 UTC
Should I knowingly put code in a test suite that intentionally makes changes to the state of a system? That depends what you mean by changing the state of a system. If it's just a matter of creating (and afterwards deleting) of files, and it's done in the tmp directory, go ahead, that's what the tmp directory is for. But if it's anything non-trivial, like opening ports, or doing something that would require root/admin priviledges, never, ever do something without 1) clearly stating to the user what you are about to do, and 2) explicitly asking the user for permission. And by all means, try to determine whether you are run from an interactive session or not, and if not, don't do the tests. Abigail	[reply]
Re: Re: Test code that makes filesystem modifications? by tovod-everett (Sexton) on Mar 04, 2004 at 17:43 UTC
It is just a matter of creating and deleting files in the tmp directory, although the following negative possibilities exist: Some sort of Ctrl-C event prevents file deletion at the end. Permissions get set to something really ugly on an object, preventing deletion until the permissions are reset to something cleaner. Since the person executing the code is the owner of all objects they create, they retain the right to change permissions on them, but my cleanup code doesn't attempt to reset permissions, especially since it'd sort of be relying on the code it's testing in the first place. I do have to write the test suite to behave differently depending upon whether the user has admin or not (and actually, ideally, the test suite would be run under both situations, which is obviously _not_ something I can fake in the test suite:). In any event, I've decided to do this: Release the .t file in some sort of disabled form. Put a message up high in the README asking for people to run the disabled test suite and let me know how it goes First off, this side steps my nervousness. Second, the test suite will probably comprise somewhere around 50,000 to 200,000 tests, and may take on the order of an hour to run! I've decided that one of the purposes of an exhaustive test suite is to be very sure there are no latent bugs in my core logic. I've found so many bugs caused by my insufficiently thorough understanding of exactly how Microsoft implements permission inheritance that I've written fairly efficient code to test the biggest conceivable envelopes. As in, here's 40 permissions. Apply every possible combination of them to a directory tree that's three deep and check that my code _never_ reports a wrong or missing inherited permission. 404040 = 64,000. I may scale this back a little:). --Toby Ovod-Everett	[reply]
Re: Test code that makes filesystem modifications? by adrianh (Chancellor) on Mar 01, 2004 at 22:49 UTC
I posted this to the modules mailing list, but the signal/noise ratio on that lists seems to have fallen through the floor, so it doesn't surprise me that I got now responses. For testing related stuff you might want to try perl-qa@perl.org which is focused on that very topic. So, should I risk this? Is this a reasonable solution? How and where should I put appropriate warnings? What Abigail said :-) Ask for the configuration information you need in your Makefile.PL/Build.PL. If you can figure it out don't ask if you're not running in an interactive session. Skip the tests if you cannot get the appropriate configuration info from the user. Module::Build has some handy functions for doing this.	[reply]
Re: Test code that makes filesystem modifications? by dragonchild (Archbishop) on Mar 02, 2004 at 02:22 UTC
Have you checked out Test::Cmd? I've used it quite successfully on Unix platforms for something similar to this. ------ We are the carpenters and bricklayers of the Information Age. Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.	[reply]