Chatbot testing using AI planning

Chatbots, i.e., systems that can interact with humans in a more appropriate way using natural language, have been of increasing importance. This is due the fact of the availability of computational means for natural language interaction between computers and humans that are becoming closer to the interaction between humans alone. Consequently, there are more and more chatbots available that are intended to support humans organizing tasks or making decisions. In this paper, we focus on how to verify the communication capabilities provided by chatbots. In particular, we introduce an automated approach for generating communication sequences and carrying them out. The approach is based on AI planning where each action can be assumed to be a certain question that is given to the chatbot. The answer of the chatbot should make the action post-condition true, in order to proceed with the plan. In cases of deviations between the actual chatbot behavior and the expected one, re-planning is required. Besides the approach, we discuss its application to the domain of tourism and outline a case study.