Test harnesses for black box tests vs. white box tests

Most unit testing test harnesses are for white-box testing. They're designed to stay out of the way and allow the programmer to write quick checks for a unit under development. If any unit test fails, the build should break and the build should fail.

When managing automation for black-box test cases against a system, the rules are different. In rare instances the tests will be ready ahead of a significant amount of feature work. Black box integration tests can rarely be complete before the functionality under test is nearly code complete, so authors of automation for black-box tests control the risks of holding up feature completeness by writing test skeletons which will fail until the test can be complete early on in feature development. Since these tests always fail, black-box test harnesses need a means to separate tests which are under development from tests which are complete.

The same goes for tests which have caught bugs: to reduce noise in a continuous deployment pipeline, these tests need to be disabled, but there is great value in running these tests in a separate workflow so that the fix can be detected by the automation.

White-box unit test needs/expectations:

• Fast individual tests
• Mocking of dependencies is always okay
• Event injection can speed up tests
• Test suites are built up gradually as the individual units are developed
• Tests often fail for obvious reasons by default
• There is no environment to gather details around for repro purposes; just re-run the tests!

Black-box integration test needs:

• Mocking of dependencies defeats the purpose of integration tests
• Asynchronous waiting for dependent services and normal system behavior is a fact of life
• Tests will often depend on one another in order to run suites in a reasonable amount of time
• Tests routinely fail for reasons different from what is explicitly under test
• Capturing environment data for a repro is necessary
• Tests are defined in combinatorial clumps rather than gradually, as new functionality is built up. Mutators, Pairwise testing, Model-based testing, and any approaches based on filling in code coverage gaps generate large numbers of test procedures which are most conveniently managed as a group.