How can you possibly test modern software fully?

From 48 to 12 Test Runs

After pairing up the variables, add a column and number the test runs for easy reference. The following matrix, Figure 24, summarises the final test runs.

Instead of 48 test runs, only 12 are required to provide the proper test coverage. If particular variable combinations have resulted in a higher number of defects in the past, you might want to include additional runs to cover those combinations.

Now, the next thing you have to do is find an automated testing tool to code up the tests and manage the test runs. [There'll be a follow-up interview with Keith in Reg Dev soon, where I'll ask him about implementing this technique in practice - David Norfolk, Ed]

Follow-up Resources

Berger, Bernie. Efficient Testing with All-Pairs International Conference on Software Testing 2003: available here.

Dustin, Elfriede. Orthogonally Speaking: A Method for deriving a suitable set of test cases STQE September/October 2001: available here.

Cohen, David M., Siddhartha R. Dalal, Jesse Parelius, and Gardner C. Patton. The Combinatorial Design Approach to Automatic Test Generation IEEE Software September 1996.

Wallace, Dolores R. and D. Richard Kuhn. Converting System Failure Histories into Future Win Situations Information Technology Laboratory, National Institute of Standards and Technology January 2000 (pdf file); there's a link to a related website here.

About the Author: Keith Vanden Eynden is a senior technical writer at Seapine Software, Inc, with 15 years experience in performance improvement, training, and documentation.