L*-Based Learning of Markov Decision Processes

Martin Tappler, Bernhard Aichernig, Giovanni Bacci, Maria Eichlseder, Kim Guldstrand Larsen

Publikation: Beitrag in Buch/Bericht/KonferenzbandBeitrag in einem KonferenzbandForschungBegutachtung

Abstract

Automata learning techniques automatically generate system models from test observations. These techniques usually fall into two categories: passive and active. Passive learning uses a predetermined data set, e.g., system logs. In contrast, active learning actively queries the system under learning, which is considered more efficient.

An influential active learning technique is Angluin’s L∗
algorithm for regular languages which inspired several generalisations from DFAs to other automata-based modelling formalisms. In this work, we study L∗-based learning of deterministic Markov decision processes, first assuming an ideal setting with perfect information. Then, we relax this assumption and present a novel learning algorithm that collects information by sampling system traces via testing. Experiments with the implementation of our sampling-based algorithm suggest that it achieves better accuracy than state-of-the-art passive learning techniques with the same amount of test data. Unlike existing learning algorithms with predefined states, our algorithm learns the complete model structure including the states.
Originalspracheenglisch
TitelFormal Methods - The Next 30 Years
Redakteure/-innenMaurice H. ter Beek, Annabelle McIver, José N. Oliveria
ErscheinungsortCham
Herausgeber (Verlag)Springer
Seiten651 - 669
Seitenumfang19
ISBN (elektronisch)978-3-030-30942-8
ISBN (Print)978-3-030-30941-1
DOIs
PublikationsstatusVeröffentlicht - 2019
Veranstaltung2019 International Symposium on Formal Methods - Porto, Portugal
Dauer: 7 Okt 201911 Okt 2019

Publikationsreihe

NameLecture Notes in Computer Science
Band11800

Konferenz

Konferenz2019 International Symposium on Formal Methods
KurztitelFM 2019
LandPortugal
OrtPorto
Zeitraum7/10/1911/10/19

Fingerprint

Learning algorithms
Sampling
Formal languages
Model structures
Testing
Experiments
Problem-Based Learning

Fields of Expertise

  • Information, Communication & Computing

Dies zitieren

Tappler, M., Aichernig, B., Bacci, G., Eichlseder, M., & Larsen, K. G. (2019). L*-Based Learning of Markov Decision Processes. in M. H. ter Beek, A. McIver, & J. N. Oliveria (Hrsg.), Formal Methods - The Next 30 Years (S. 651 - 669). (Lecture Notes in Computer Science; Band 11800). Cham: Springer. https://doi.org/10.1007/978-3-030-30942-8_38

L*-Based Learning of Markov Decision Processes. / Tappler, Martin; Aichernig, Bernhard; Bacci, Giovanni; Eichlseder, Maria; Larsen, Kim Guldstrand.

Formal Methods - The Next 30 Years . Hrsg. / Maurice H. ter Beek; Annabelle McIver; José N. Oliveria. Cham : Springer, 2019. S. 651 - 669 (Lecture Notes in Computer Science; Band 11800).

Publikation: Beitrag in Buch/Bericht/KonferenzbandBeitrag in einem KonferenzbandForschungBegutachtung

Tappler, M, Aichernig, B, Bacci, G, Eichlseder, M & Larsen, KG 2019, L*-Based Learning of Markov Decision Processes. in MH ter Beek, A McIver & JN Oliveria (Hrsg.), Formal Methods - The Next 30 Years . Lecture Notes in Computer Science, Bd. 11800, Springer, Cham, S. 651 - 669, Porto, Portugal, 7/10/19. https://doi.org/10.1007/978-3-030-30942-8_38
Tappler M, Aichernig B, Bacci G, Eichlseder M, Larsen KG. L*-Based Learning of Markov Decision Processes. in ter Beek MH, McIver A, Oliveria JN, Hrsg., Formal Methods - The Next 30 Years . Cham: Springer. 2019. S. 651 - 669. (Lecture Notes in Computer Science). https://doi.org/10.1007/978-3-030-30942-8_38
Tappler, Martin ; Aichernig, Bernhard ; Bacci, Giovanni ; Eichlseder, Maria ; Larsen, Kim Guldstrand. / L*-Based Learning of Markov Decision Processes. Formal Methods - The Next 30 Years . Hrsg. / Maurice H. ter Beek ; Annabelle McIver ; José N. Oliveria. Cham : Springer, 2019. S. 651 - 669 (Lecture Notes in Computer Science).
@inproceedings{5e308319b0454fd4b6373238239c3238,
title = "L*-Based Learning of Markov Decision Processes",
abstract = "Automata learning techniques automatically generate system models from test observations. These techniques usually fall into two categories: passive and active. Passive learning uses a predetermined data set, e.g., system logs. In contrast, active learning actively queries the system under learning, which is considered more efficient.An influential active learning technique is Angluin’s L∗algorithm for regular languages which inspired several generalisations from DFAs to other automata-based modelling formalisms. In this work, we study L∗-based learning of deterministic Markov decision processes, first assuming an ideal setting with perfect information. Then, we relax this assumption and present a novel learning algorithm that collects information by sampling system traces via testing. Experiments with the implementation of our sampling-based algorithm suggest that it achieves better accuracy than state-of-the-art passive learning techniques with the same amount of test data. Unlike existing learning algorithms with predefined states, our algorithm learns the complete model structure including the states.",
author = "Martin Tappler and Bernhard Aichernig and Giovanni Bacci and Maria Eichlseder and Larsen, {Kim Guldstrand}",
year = "2019",
doi = "10.1007/978-3-030-30942-8_38",
language = "English",
isbn = "978-3-030-30941-1",
series = "Lecture Notes in Computer Science",
publisher = "Springer",
pages = "651 -- 669",
editor = "{ter Beek}, {Maurice H.} and Annabelle McIver and Oliveria, {Jos{\'e} N.}",
booktitle = "Formal Methods - The Next 30 Years",

}

TY - GEN

T1 - L*-Based Learning of Markov Decision Processes

AU - Tappler, Martin

AU - Aichernig, Bernhard

AU - Bacci, Giovanni

AU - Eichlseder, Maria

AU - Larsen, Kim Guldstrand

PY - 2019

Y1 - 2019

N2 - Automata learning techniques automatically generate system models from test observations. These techniques usually fall into two categories: passive and active. Passive learning uses a predetermined data set, e.g., system logs. In contrast, active learning actively queries the system under learning, which is considered more efficient.An influential active learning technique is Angluin’s L∗algorithm for regular languages which inspired several generalisations from DFAs to other automata-based modelling formalisms. In this work, we study L∗-based learning of deterministic Markov decision processes, first assuming an ideal setting with perfect information. Then, we relax this assumption and present a novel learning algorithm that collects information by sampling system traces via testing. Experiments with the implementation of our sampling-based algorithm suggest that it achieves better accuracy than state-of-the-art passive learning techniques with the same amount of test data. Unlike existing learning algorithms with predefined states, our algorithm learns the complete model structure including the states.

AB - Automata learning techniques automatically generate system models from test observations. These techniques usually fall into two categories: passive and active. Passive learning uses a predetermined data set, e.g., system logs. In contrast, active learning actively queries the system under learning, which is considered more efficient.An influential active learning technique is Angluin’s L∗algorithm for regular languages which inspired several generalisations from DFAs to other automata-based modelling formalisms. In this work, we study L∗-based learning of deterministic Markov decision processes, first assuming an ideal setting with perfect information. Then, we relax this assumption and present a novel learning algorithm that collects information by sampling system traces via testing. Experiments with the implementation of our sampling-based algorithm suggest that it achieves better accuracy than state-of-the-art passive learning techniques with the same amount of test data. Unlike existing learning algorithms with predefined states, our algorithm learns the complete model structure including the states.

U2 - 10.1007/978-3-030-30942-8_38

DO - 10.1007/978-3-030-30942-8_38

M3 - Conference contribution

SN - 978-3-030-30941-1

T3 - Lecture Notes in Computer Science

SP - 651

EP - 669

BT - Formal Methods - The Next 30 Years

A2 - ter Beek, Maurice H.

A2 - McIver, Annabelle

A2 - Oliveria, José N.

PB - Springer

CY - Cham

ER -