Comparing the fault detection effectiveness of n-way and random test suites
- 1 January 2004
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
Abstract
Software testing plays a critical role in the timely delivery of high-quality software systems. Despite the important role that testing plays, little is known about the fault detection effectiveness of many testing techniques. We investigate "n-way" test suites created using a common greedy algorithm for use in combinatorial testing. A controlled study is designed and executed to compare the fault detection effectiveness of n-way and random test suites. Combinatorial testing is conducted on target systems that have been injected with software faults. The results are that there is no significant difference in the fault detection effectiveness of n-way and random test suites for the applications studied. Analysis of the random test suite finds that they are very similar to n-way test suites from the perspective of the number of test data combinations covered. This result concurs with other hypothetical results that indicate little difference between n-way and random test suites. While we do not expect this result to apply in all combinatorial testing situations, we believe the result will lead to the design of better combinatorial test suites.Keywords
This publication has 23 references indexed in Scilit:
- Test Adequacy And Program MutationPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2005
- A practical strategy for testing pair-wise coverage of network interfacesPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2002
- Improved quality of protocol testing through techniques of experimental designPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2002
- Model-based testing in practicePublished by Association for Computing Machinery (ACM) ,1999
- Factor-Covering Designs for Testing SoftwareTechnometrics, 1998
- The AETG system: an approach to testing based on combinatorial designIEEE Transactions on Software Engineering, 1997
- Applying design of experiments to software testingPublished by Association for Computing Machinery (ACM) ,1997
- Robust Testing of AT&T PMX/StarMAIL Using OATSAT&T Technical Journal, 1992
- Investigations of the software testing coupling effectACM Transactions on Software Engineering and Methodology, 1992
- Orthogonal Latin squares: an application of experiment design to compiler testingCommunications of the ACM, 1985