Mutation Score, Coverage, Model Inference: Quality Assessment for T-Way Combinatorial Test-Suites

Hermann Felbinger, Franz Wotawa, Mihai Nica

Publikation: Beitrag in Buch/Bericht/KonferenzbandBeitrag in einem KonferenzbandBegutachtung


In this paper we assess and evaluate the quality of t-way combinatorial test-suites using three different test-suite quality assessment methods. As t-way combinatorial test-suites reduce the input space of a program under test, we investigate how an increasing t affects the quality of the test-suite. There are some limitations of existing test-suite quality assessment methods e.g. the number of mutants is limited by execution time and code coverage measurement might be intrusive due to changes of the behavior of the program under test when instrumenting the code. Here we generate t-way combinatorial test-suites for Java programs of different size. We compute mutation score and code coverage for the generated test-suites, and apply additionally a new model inference based approach, that does not require to execute the program under test, to compare the generated test-suites with each other and assign a quality valuation to the test-suites. Our results show that an increasing t generally raises test-suite quality in terms of mutation score, coverage, and model inference. However, the model inference approach is only applicable, if the outcomes of the programs under test are discrete values, and if the number of discrete values is less than the test-suite size.

TitelProceedings - 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017
Herausgeber (Verlag)Institute of Electrical and Electronics Engineers
ISBN (elektronisch)978-1-5090-6676-6
PublikationsstatusVeröffentlicht - 13 Apr 2017
Veranstaltung10th IEEE International Conference on Software Testing, Verification and Validation Workshops: ICSTW 2017 - Tokyo, Japan
Dauer: 13 Mär 201717 Mär 2017


Konferenz10th IEEE International Conference on Software Testing, Verification and Validation Workshops

ASJC Scopus subject areas

  • Software
  • Sicherheit, Risiko, Zuverlässigkeit und Qualität

Dieses zitieren