Mutation Score, Coverage, Model Inference: Quality Assessment for T-Way Combinatorial Test-Suites

Hermann Felbinger, Franz Wotawa, Mihai Nica

Publikation: Beitrag in Buch/Bericht/KonferenzbandBeitrag in einem KonferenzbandForschungBegutachtung

Abstract

In this paper we assess and evaluate the quality of t-way combinatorial test-suites using three different test-suite quality assessment methods. As t-way combinatorial test-suites reduce the input space of a program under test, we investigate how an increasing t affects the quality of the test-suite. There are some limitations of existing test-suite quality assessment methods e.g. the number of mutants is limited by execution time and code coverage measurement might be intrusive due to changes of the behavior of the program under test when instrumenting the code. Here we generate t-way combinatorial test-suites for Java programs of different size. We compute mutation score and code coverage for the generated test-suites, and apply additionally a new model inference based approach, that does not require to execute the program under test, to compare the generated test-suites with each other and assign a quality valuation to the test-suites. Our results show that an increasing t generally raises test-suite quality in terms of mutation score, coverage, and model inference. However, the model inference approach is only applicable, if the outcomes of the programs under test are discrete values, and if the number of discrete values is less than the test-suite size.

Originalspracheenglisch
TitelProceedings - 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017
Herausgeber (Verlag)Institute of Electrical and Electronics Engineers
Seiten171-180
Seitenumfang10
ISBN (elektronisch)978-1-5090-6676-6
DOIs
PublikationsstatusVeröffentlicht - 13 Apr 2017
Veranstaltung10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017 - Tokyo, Japan
Dauer: 13 Mär 201717 Mär 2017

Konferenz

Konferenz10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017
LandJapan
OrtTokyo
Zeitraum13/03/1717/03/17

Schlagwörter

    ASJC Scopus subject areas

    • Software
    • !!Safety, Risk, Reliability and Quality

    Dies zitieren

    Felbinger, H., Wotawa, F., & Nica, M. (2017). Mutation Score, Coverage, Model Inference: Quality Assessment for T-Way Combinatorial Test-Suites. in Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017 (S. 171-180). [7899053] Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ICSTW.2017.36

    Mutation Score, Coverage, Model Inference : Quality Assessment for T-Way Combinatorial Test-Suites. / Felbinger, Hermann; Wotawa, Franz; Nica, Mihai.

    Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017. Institute of Electrical and Electronics Engineers, 2017. S. 171-180 7899053.

    Publikation: Beitrag in Buch/Bericht/KonferenzbandBeitrag in einem KonferenzbandForschungBegutachtung

    Felbinger, H, Wotawa, F & Nica, M 2017, Mutation Score, Coverage, Model Inference: Quality Assessment for T-Way Combinatorial Test-Suites. in Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017., 7899053, Institute of Electrical and Electronics Engineers, S. 171-180, Tokyo, Japan, 13/03/17. https://doi.org/10.1109/ICSTW.2017.36
    Felbinger H, Wotawa F, Nica M. Mutation Score, Coverage, Model Inference: Quality Assessment for T-Way Combinatorial Test-Suites. in Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017. Institute of Electrical and Electronics Engineers. 2017. S. 171-180. 7899053 https://doi.org/10.1109/ICSTW.2017.36
    Felbinger, Hermann ; Wotawa, Franz ; Nica, Mihai. / Mutation Score, Coverage, Model Inference : Quality Assessment for T-Way Combinatorial Test-Suites. Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017. Institute of Electrical and Electronics Engineers, 2017. S. 171-180
    @inproceedings{d8dad5f03a4745ad99a5f0e51cacbdc0,
    title = "Mutation Score, Coverage, Model Inference: Quality Assessment for T-Way Combinatorial Test-Suites",
    abstract = "In this paper we assess and evaluate the quality of t-way combinatorial test-suites using three different test-suite quality assessment methods. As t-way combinatorial test-suites reduce the input space of a program under test, we investigate how an increasing t affects the quality of the test-suite. There are some limitations of existing test-suite quality assessment methods e.g. the number of mutants is limited by execution time and code coverage measurement might be intrusive due to changes of the behavior of the program under test when instrumenting the code. Here we generate t-way combinatorial test-suites for Java programs of different size. We compute mutation score and code coverage for the generated test-suites, and apply additionally a new model inference based approach, that does not require to execute the program under test, to compare the generated test-suites with each other and assign a quality valuation to the test-suites. Our results show that an increasing t generally raises test-suite quality in terms of mutation score, coverage, and model inference. However, the model inference approach is only applicable, if the outcomes of the programs under test are discrete values, and if the number of discrete values is less than the test-suite size.",
    keywords = "Combinatorial testing, Coverage, Model inference, Mutation score",
    author = "Hermann Felbinger and Franz Wotawa and Mihai Nica",
    year = "2017",
    month = "4",
    day = "13",
    doi = "10.1109/ICSTW.2017.36",
    language = "English",
    pages = "171--180",
    booktitle = "Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017",
    publisher = "Institute of Electrical and Electronics Engineers",
    address = "United States",

    }

    TY - GEN

    T1 - Mutation Score, Coverage, Model Inference

    T2 - Quality Assessment for T-Way Combinatorial Test-Suites

    AU - Felbinger, Hermann

    AU - Wotawa, Franz

    AU - Nica, Mihai

    PY - 2017/4/13

    Y1 - 2017/4/13

    N2 - In this paper we assess and evaluate the quality of t-way combinatorial test-suites using three different test-suite quality assessment methods. As t-way combinatorial test-suites reduce the input space of a program under test, we investigate how an increasing t affects the quality of the test-suite. There are some limitations of existing test-suite quality assessment methods e.g. the number of mutants is limited by execution time and code coverage measurement might be intrusive due to changes of the behavior of the program under test when instrumenting the code. Here we generate t-way combinatorial test-suites for Java programs of different size. We compute mutation score and code coverage for the generated test-suites, and apply additionally a new model inference based approach, that does not require to execute the program under test, to compare the generated test-suites with each other and assign a quality valuation to the test-suites. Our results show that an increasing t generally raises test-suite quality in terms of mutation score, coverage, and model inference. However, the model inference approach is only applicable, if the outcomes of the programs under test are discrete values, and if the number of discrete values is less than the test-suite size.

    AB - In this paper we assess and evaluate the quality of t-way combinatorial test-suites using three different test-suite quality assessment methods. As t-way combinatorial test-suites reduce the input space of a program under test, we investigate how an increasing t affects the quality of the test-suite. There are some limitations of existing test-suite quality assessment methods e.g. the number of mutants is limited by execution time and code coverage measurement might be intrusive due to changes of the behavior of the program under test when instrumenting the code. Here we generate t-way combinatorial test-suites for Java programs of different size. We compute mutation score and code coverage for the generated test-suites, and apply additionally a new model inference based approach, that does not require to execute the program under test, to compare the generated test-suites with each other and assign a quality valuation to the test-suites. Our results show that an increasing t generally raises test-suite quality in terms of mutation score, coverage, and model inference. However, the model inference approach is only applicable, if the outcomes of the programs under test are discrete values, and if the number of discrete values is less than the test-suite size.

    KW - Combinatorial testing

    KW - Coverage

    KW - Model inference

    KW - Mutation score

    UR - http://www.scopus.com/inward/record.url?scp=85018371142&partnerID=8YFLogxK

    U2 - 10.1109/ICSTW.2017.36

    DO - 10.1109/ICSTW.2017.36

    M3 - Conference contribution

    SP - 171

    EP - 180

    BT - Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017

    PB - Institute of Electrical and Electronics Engineers

    ER -