Mutation Score, Coverage, Model Inference: Quality Assessment for T-Way Combinatorial Test-Suites

Hermann Felbinger, Franz Wotawa, Mihai Nica

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

Abstract

In this paper we assess and evaluate the quality of t-way combinatorial test-suites using three different test-suite quality assessment methods. As t-way combinatorial test-suites reduce the input space of a program under test, we investigate how an increasing t affects the quality of the test-suite. There are some limitations of existing test-suite quality assessment methods e.g. the number of mutants is limited by execution time and code coverage measurement might be intrusive due to changes of the behavior of the program under test when instrumenting the code. Here we generate t-way combinatorial test-suites for Java programs of different size. We compute mutation score and code coverage for the generated test-suites, and apply additionally a new model inference based approach, that does not require to execute the program under test, to compare the generated test-suites with each other and assign a quality valuation to the test-suites. Our results show that an increasing t generally raises test-suite quality in terms of mutation score, coverage, and model inference. However, the model inference approach is only applicable, if the outcomes of the programs under test are discrete values, and if the number of discrete values is less than the test-suite size.

Original languageEnglish
Title of host publicationProceedings - 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017
PublisherInstitute of Electrical and Electronics Engineers
Pages171-180
Number of pages10
ISBN (Electronic)978-1-5090-6676-6
DOIs
Publication statusPublished - 13 Apr 2017
Event10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017 - Tokyo, Japan
Duration: 13 Mar 201717 Mar 2017

Conference

Conference10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017
CountryJapan
CityTokyo
Period13/03/1717/03/17

Keywords

  • Combinatorial testing
  • Coverage
  • Model inference
  • Mutation score

ASJC Scopus subject areas

  • Software
  • Safety, Risk, Reliability and Quality

Cite this

Felbinger, H., Wotawa, F., & Nica, M. (2017). Mutation Score, Coverage, Model Inference: Quality Assessment for T-Way Combinatorial Test-Suites. In Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017 (pp. 171-180). [7899053] Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ICSTW.2017.36