moot has asked for the wisdom of the Perl Monks concerning the following question:

I'm refactoring a bunch of code that needs to determine currently available network interfaces and will be using IO::Interface to do the heavy lifting.

As part of the refactor I'm writing tests to ensure that interfaces for the system under test are detected correctly. These are obviously system-dependent.

The test suite for IO::Interface simply looks for a local loopback. What I'd like to do is somehow feed my test suite a list of expected interfaces, along with IP, broadcast, ether address, etc for each one.

Specifically, what would be the best practice for this? Seems my options are:

  1. Have the tester write the info to a file which the test script can read and parse (which opens up a whole 'nuther can of wirms).
  2. Have the tester pass the info to the test script. Again, parsing the info will be an issue.
  3. Have the test script bypass my code and use IO::Interface directly to establish the interfaces, and compare these with the output from my module's methods.
  4. Have the test script query the user at test-time. Again, parsing input, and also not very automatic.
  5. Something Else.

Seems options 1,2, and 4 are all variants of the same thing and have too many issues to be useful. Mainly, automated testing would be broken for most systems. Unless "Something Else" involves some magic of which I am unaware, then, the best course of action would be the third option. Of course, if I weren't using IO::Interface I would have to establish another solution.

So much for this specific problem, then, but it raises the general question of how to specify system-dependent information to test scripts in an automated way.

I've super-searched and googled but results were unsatisfactory, although that's probably the fault of my search terms rather than because no-one has encountered this issue before.

In the past I've managed to avoid this question by running manually all tests for which I have responsibility, and I'm therefore able to control the environment. As part of a more outward-thinking software development effort though I'd be interested to learn of any approach I may be missing. I'm sure this question must come up regularly so if there's an obvious approach I'm missing, feel free to smack me.

  • Comment on IO::Interface and System-dependent Testing

Replies are listed 'Best First'.
Re: IO::Interface and System-dependent Testing
by kvale (Monsignor) on Apr 20, 2005 at 18:15 UTC
    I think it is certainly reasonable to automatically test for system dependences before running the test suite. In your case, you need to ensure that the network is up and that certain services are available and listening, etc. As a part of this setup, you will learn IP and ethernet addresses that could be used in the test suite.

    Once the initial setup is done, however, it would be best to arrange your test suite so that tests do not rely on any other system dependent stuff other than what was gathered during setup. The point of testing is to validate known output against known input, so reporoducibility is key and the actual test suite should be as machine independent as possible. This sort of attitude also helps you to eliminate implicit machine dependencies in the code you are testing.

    -Mark

[reply]
Re: IO::Interface and System-dependent Testing
by starbolin (Hermit) on Apr 20, 2005 at 23:00 UTC

    It sounds like your code performs some Session Management and uses IO::Interface to interface down into the Network Layer. Now if your testing relies on whatever Network Layer is default on the machine then you don't have real independance. Since a given machine has a limited and maybe fixed set of interfaces you also have no way to ensure coverage in you detection code.

    The solution is to provide interface stubs to be used for testing. Eth0, tu0 et. al. could then be written to provide system indepent behavior and also suitable edge cases.

    s//----->\t/;$~="JAPH";s//\r<$~~/;{s|~$~-|-~$~|||s |-$~~|$~~-|||s,<$~~,<~$~,,s,~$~>,$~~>,, $|=1,select$,,$,,$,,1e-1;print;redo}
[reply]
[d/l]
      Interesting response. I have no idea what led you to the conclusion that the code performs session management..

      I'm not too worried that the code relies on layer 2 or layer 3 information, that much is a given for the nature of the work the code is doing. What concerns me is that on linux I may need to know eth0 is 'external' and eth1 'internal', where on BSD these may be combinations of en, xl, or fxp. The actual name of the interface is secondary; it's the IP and netmask that are more important.

      Likewise, the detection of the interfaces is secondary; I just need to have something on which to base other higher-level tests, which needs to be valid for the system under scrutiny.

      As I said I can use IO::Interface to verify that my code is doing the right thing (my module just extends IO::Interface), so in this particular instance, the problem is solved.

      The bigger question is how to provide some level of system independence while interrogating the system for information (whether it's network interfaces, ARP table entries, or the partition table of a drive), and have this done in an automated manner so non-technical QA staff can run unit tests as part of their code verification.

      I realise the target is somewhat vague, but shurely shomeone shomewhere has run into this before.

[reply]

        moot writes:

        I have no idea what led you to the conclusion that the code performs session management.
        this, from OP:
        ,,,bunch of code that needs to determine currently available network interfaces

        moot writes:

        The actual name of the interface is secondary; it's the IP and netmask that are more important.
        And each is a different level of abstration, a different level of complexity. The deaper you go into the the layers the more information you have to provide and the less isolation you have from the enviroment. The distictions between layers are not "secondary"; they are the core of the problem.

        To rephrase my original post. If we're testing a database application we provide an artificial dataset for an environment. Why wouldn't we provide an artificial environment for a network applicaiton?

        I'm sorry I misunderstood your OP and I know I'm not addressing your core issue of automation but please don't shoot the messenger. As you said yourself:"the target is somewhat vague".

        s//----->\t/;$~="JAPH";s//\r<$~~/;{s|~$~-|-~$~|||s |-$~~|$~~-|||s,<$~~,<~$~,,s,~$~>,$~~>,, $|=1,select$,,$,,$,,1e-1;print;redo}
[reply]

Back to Seekers of Perl Wisdom