Fully automated compiler testing of a reasoning engine via mutated grammar fuzzing

Yavuz Koroglu, Franz Wotawa

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

Abstract

A reasoning engine infers logical consequences from a set of fixed axioms and observations. However, before it can make an inference, it must compile the axioms and observations which are given in a predefined format. Any attempt to test the correctness of a reasoning engine assumes that it compiles inputs correctly, but that may not be the case. In this work, we implement a mutated grammar fuzzer to automatically generate tests for the compilation stage of Assumption-based Truth Maintenance System (ATMS), a reasoning engine for model-based diagnosis. We also implement a recognizer as an oracle and automatically evaluate the correctness of compiler output. We automatically generate, execute, and evaluate more than a million tests in two weeks. We show that while tests generated from the true grammar of ATMS find no faults, tests generated from mutated grammars uncover an important fault in the compiler. We also show that mutated grammars achieve higher code coverage with fewer tests and the original grammar cannot cover any code that is not covered by mutated grammars. To the best of our knowledge, ours is the first work that provides a practical implementation and evaluation of a mutated grammar fuzzer. We make the implementation available online along with small examples, tests generated for this paper, and steps to reproduce our experiments.

Original languageEnglish
Title of host publicationProceedings - 2019 IEEE/ACM 14th International Workshop on Automation of Software Test, AST 2019
PublisherInstitute of Electrical and Electronics Engineers
Pages28-34
Number of pages7
ISBN (Electronic)9781728122373
DOIs
Publication statusPublished - 1 May 2019
Event14th IEEE/ACM International Workshop on Automation of Software Test, AST 2019 - Montreal, Canada
Duration: 27 May 2019 → …

Publication series

NameProceedings - 2019 IEEE/ACM 14th International Workshop on Automation of Software Test, AST 2019

Conference

Conference14th IEEE/ACM International Workshop on Automation of Software Test, AST 2019
CountryCanada
CityMontreal
Period27/05/19 → …

    Fingerprint

Keywords

  • Compiler testing
  • Fuzz testing
  • Grammar fuzzing
  • Mutation testing

ASJC Scopus subject areas

  • Software
  • Safety, Risk, Reliability and Quality
  • Control and Optimization

Cite this

Koroglu, Y., & Wotawa, F. (2019). Fully automated compiler testing of a reasoning engine via mutated grammar fuzzing. In Proceedings - 2019 IEEE/ACM 14th International Workshop on Automation of Software Test, AST 2019 (pp. 28-34). [8822051] (Proceedings - 2019 IEEE/ACM 14th International Workshop on Automation of Software Test, AST 2019). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/AST.2019.00010