Extending Automated FLTL Test Oracles With Diagnostic Support

Testing is a versatile and in practice also dominant technique when it comes to verifying whether a system meets our expectations. After executing a test case, we use test oracles to judge whether the execution should be considered to have failed or passed. Fully automated oracles considering properties in temporal logics like FLTL allow us to derive such a verdict in a fully automated process. In this manuscript, we will show how to extend such an oracle with diagnostic support. In particular, drawing on model-based diagnosis (MBD), we will isolate exactly which parts of the property were violated for a failed test case. Such data are orthogonal to MBD focusing on the system itself and where we isolate faulty system components. With our diagnoses, we thus provide valuable information for the subsequent debugging and repair process in respect of how the test execution violated the property. We show that a corresponding polynomially sized SAT model for deriving our diagnoses can be derived easily.