Embodied synaptic plasticity with online reinforcement learning

Jacques Kaiser, Michael Hoff, Andreas Konle, J. Camilo Vasquez Tieck, David Kappel, Daniel Reichard, Anand Subramoney, Robert Legenstein, Arne Roennau, Wolfgang Maass, Rüdiger Dillmann

Publikation: Beitrag in einer FachzeitschriftArtikelForschungBegutachtung

Abstract

The endeavor to understand the brain involves multiple collaborating research fields. Classically, synaptic plasticity rules derived by theoretical neuroscientists are evaluated in isolation on pattern classification tasks. This contrasts with the biological brain which purpose is to control a body in closed-loop. This paper contributes to bringing the fields of computational neuroscience and robotics closer together by integrating open-source software components fromthese two fields. The resulting framework allows to evaluate the validity of biologically-plausibe plasticity models in closed-loop robotics environments. We demonstrate this framework to evaluate Synaptic Plasticity with Online REinforcement learning (SPORE), a reward-learning rule based on synaptic sampling, on two visuomotor tasks: reaching and lane following. We show that SPORE is capable of learning to perform policies within the course of simulated hours for both tasks. Provisional parameter explorations indicate that the learning rate and the temperature driving the stochastic processes that govern synaptic learning dynamics need to be regulated for performance improvements to be retained. We conclude by discussing the recent deep reinforcement learning techniques which would be beneficial to increase the functionality of SPORE on visuomotor tasks.

Originalspracheenglisch
Aufsatznummer81
Seiten (von - bis)1-11
Seitenumfang11
FachzeitschriftFrontiers in Neurorobotics
Jahrgang13
DOIs
PublikationsstatusVeröffentlicht - 3 Okt 2019

Fingerprint

Reinforcement learning
Plasticity
Brain
Robotics
Random processes
Pattern recognition
Sampling
Temperature

Schlagwörter

    ASJC Scopus subject areas

    • !!Biomedical Engineering
    • Artificial intelligence

    Dies zitieren

    Kaiser, J., Hoff, M., Konle, A., Tieck, J. C. V., Kappel, D., Reichard, D., ... Dillmann, R. (2019). Embodied synaptic plasticity with online reinforcement learning. Frontiers in Neurorobotics, 13, 1-11. [81]. https://doi.org/10.3389/fnbot.2019.00081

    Embodied synaptic plasticity with online reinforcement learning. / Kaiser, Jacques; Hoff, Michael; Konle, Andreas; Tieck, J. Camilo Vasquez; Kappel, David; Reichard, Daniel; Subramoney, Anand; Legenstein, Robert; Roennau, Arne; Maass, Wolfgang; Dillmann, Rüdiger.

    in: Frontiers in Neurorobotics, Jahrgang 13, 81, 03.10.2019, S. 1-11.

    Publikation: Beitrag in einer FachzeitschriftArtikelForschungBegutachtung

    Kaiser, J, Hoff, M, Konle, A, Tieck, JCV, Kappel, D, Reichard, D, Subramoney, A, Legenstein, R, Roennau, A, Maass, W & Dillmann, R 2019, 'Embodied synaptic plasticity with online reinforcement learning' Frontiers in Neurorobotics, Jg. 13, 81, S. 1-11. https://doi.org/10.3389/fnbot.2019.00081
    Kaiser J, Hoff M, Konle A, Tieck JCV, Kappel D, Reichard D et al. Embodied synaptic plasticity with online reinforcement learning. Frontiers in Neurorobotics. 2019 Okt 3;13:1-11. 81. https://doi.org/10.3389/fnbot.2019.00081
    Kaiser, Jacques ; Hoff, Michael ; Konle, Andreas ; Tieck, J. Camilo Vasquez ; Kappel, David ; Reichard, Daniel ; Subramoney, Anand ; Legenstein, Robert ; Roennau, Arne ; Maass, Wolfgang ; Dillmann, Rüdiger. / Embodied synaptic plasticity with online reinforcement learning. in: Frontiers in Neurorobotics. 2019 ; Jahrgang 13. S. 1-11.
    @article{4d1a97597d3240a29942cabdf968bda5,
    title = "Embodied synaptic plasticity with online reinforcement learning",
    abstract = "The endeavor to understand the brain involves multiple collaborating research fields. Classically, synaptic plasticity rules derived by theoretical neuroscientists are evaluated in isolation on pattern classification tasks. This contrasts with the biological brain which purpose is to control a body in closed-loop. This paper contributes to bringing the fields of computational neuroscience and robotics closer together by integrating open-source software components fromthese two fields. The resulting framework allows to evaluate the validity of biologically-plausibe plasticity models in closed-loop robotics environments. We demonstrate this framework to evaluate Synaptic Plasticity with Online REinforcement learning (SPORE), a reward-learning rule based on synaptic sampling, on two visuomotor tasks: reaching and lane following. We show that SPORE is capable of learning to perform policies within the course of simulated hours for both tasks. Provisional parameter explorations indicate that the learning rate and the temperature driving the stochastic processes that govern synaptic learning dynamics need to be regulated for performance improvements to be retained. We conclude by discussing the recent deep reinforcement learning techniques which would be beneficial to increase the functionality of SPORE on visuomotor tasks.",
    keywords = "Neuromorphic vision, Neurorobotics, Reinforcement learning, Spiking neural networks, Synaptic plasticity",
    author = "Jacques Kaiser and Michael Hoff and Andreas Konle and Tieck, {J. Camilo Vasquez} and David Kappel and Daniel Reichard and Anand Subramoney and Robert Legenstein and Arne Roennau and Wolfgang Maass and R{\"u}diger Dillmann",
    year = "2019",
    month = "10",
    day = "3",
    doi = "10.3389/fnbot.2019.00081",
    language = "English",
    volume = "13",
    pages = "1--11",
    journal = "Frontiers in Neurorobotics",
    issn = "1662-5218",
    publisher = "Frontiers Research Foundation",

    }

    TY - JOUR

    T1 - Embodied synaptic plasticity with online reinforcement learning

    AU - Kaiser, Jacques

    AU - Hoff, Michael

    AU - Konle, Andreas

    AU - Tieck, J. Camilo Vasquez

    AU - Kappel, David

    AU - Reichard, Daniel

    AU - Subramoney, Anand

    AU - Legenstein, Robert

    AU - Roennau, Arne

    AU - Maass, Wolfgang

    AU - Dillmann, Rüdiger

    PY - 2019/10/3

    Y1 - 2019/10/3

    N2 - The endeavor to understand the brain involves multiple collaborating research fields. Classically, synaptic plasticity rules derived by theoretical neuroscientists are evaluated in isolation on pattern classification tasks. This contrasts with the biological brain which purpose is to control a body in closed-loop. This paper contributes to bringing the fields of computational neuroscience and robotics closer together by integrating open-source software components fromthese two fields. The resulting framework allows to evaluate the validity of biologically-plausibe plasticity models in closed-loop robotics environments. We demonstrate this framework to evaluate Synaptic Plasticity with Online REinforcement learning (SPORE), a reward-learning rule based on synaptic sampling, on two visuomotor tasks: reaching and lane following. We show that SPORE is capable of learning to perform policies within the course of simulated hours for both tasks. Provisional parameter explorations indicate that the learning rate and the temperature driving the stochastic processes that govern synaptic learning dynamics need to be regulated for performance improvements to be retained. We conclude by discussing the recent deep reinforcement learning techniques which would be beneficial to increase the functionality of SPORE on visuomotor tasks.

    AB - The endeavor to understand the brain involves multiple collaborating research fields. Classically, synaptic plasticity rules derived by theoretical neuroscientists are evaluated in isolation on pattern classification tasks. This contrasts with the biological brain which purpose is to control a body in closed-loop. This paper contributes to bringing the fields of computational neuroscience and robotics closer together by integrating open-source software components fromthese two fields. The resulting framework allows to evaluate the validity of biologically-plausibe plasticity models in closed-loop robotics environments. We demonstrate this framework to evaluate Synaptic Plasticity with Online REinforcement learning (SPORE), a reward-learning rule based on synaptic sampling, on two visuomotor tasks: reaching and lane following. We show that SPORE is capable of learning to perform policies within the course of simulated hours for both tasks. Provisional parameter explorations indicate that the learning rate and the temperature driving the stochastic processes that govern synaptic learning dynamics need to be regulated for performance improvements to be retained. We conclude by discussing the recent deep reinforcement learning techniques which would be beneficial to increase the functionality of SPORE on visuomotor tasks.

    KW - Neuromorphic vision

    KW - Neurorobotics

    KW - Reinforcement learning

    KW - Spiking neural networks

    KW - Synaptic plasticity

    UR - http://www.scopus.com/inward/record.url?scp=85073797479&partnerID=8YFLogxK

    U2 - 10.3389/fnbot.2019.00081

    DO - 10.3389/fnbot.2019.00081

    M3 - Article

    VL - 13

    SP - 1

    EP - 11

    JO - Frontiers in Neurorobotics

    JF - Frontiers in Neurorobotics

    SN - 1662-5218

    M1 - 81

    ER -