Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype

Chen Liu, Guillaume Emmanuel Fernand Bellec, Bernhard Vogginger, David Kappel, Johannes Partzsch, Felix Neumärker, Sebastian Höppner, Wolfgang Maass, Steve B. Furber, Robert Legenstein, Christian G. Mayr

Publikation: Beitrag in einer FachzeitschriftArtikelForschungBegutachtung

Abstract

The memory requirement of deep learning algorithms is considered incompatible with the memory restriction of energy-efficient hardware. A low memory footprint can be achieved by pruning obsolete connections or reducing the precision of connection strengths after the network has been trained. Yet, these techniques are not applicable to the case when neural networks have to be trained directly on hardware due to the hard memory constraints. Deep Rewiring (DEEP R) is a training algorithm which continuously rewires the network while preserving very sparse connectivity all along the training procedure.We apply DEEP R to a deep neural network implementation on a prototype chip of the 2nd
generation SpiNNaker system. The local memory of a single core on this chip is limited to 64 KB and a deep network architecture is trained entirely within this constraint without the use of external memory. Throughout training, the proportion of active connections is limited to 1.3%. On the handwritten digits dataset MNIST, this extremely sparse network achieves 96.6% classification accuracy at convergence. Utilizing the multi-processor feature of the SpiNNaker system, we found very good scaling in terms of computation time, per-core memory consumption, and energy constraints. When compared to a X86 CPU implementation, neural network training on the SpiNNaker 2 prototype improves
power and energy consumption by two orders of magnitude.
Originalspracheenglisch
Aufsatznummer840
Seitenumfang15
FachzeitschriftFrontiers in neuroscience
Jahrgang12
Ausgabenummer840
DOIs
PublikationsstatusVeröffentlicht - 19 Nov 2018

Fingerprint

Learning
Datasets

Schlagwörter

    Dies zitieren

    Liu, C., Bellec, G. E. F., Vogginger, B., Kappel, D., Partzsch, J., Neumärker, F., ... Mayr, C. G. (2018). Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype. Frontiers in neuroscience, 12(840), [840]. https://doi.org/10.3389/fnins.2018.00840

    Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype. / Liu, Chen; Bellec, Guillaume Emmanuel Fernand; Vogginger, Bernhard; Kappel, David; Partzsch, Johannes; Neumärker, Felix; Höppner, Sebastian; Maass, Wolfgang; Furber, Steve B.; Legenstein, Robert; Mayr, Christian G.

    in: Frontiers in neuroscience, Jahrgang 12, Nr. 840, 840, 19.11.2018.

    Publikation: Beitrag in einer FachzeitschriftArtikelForschungBegutachtung

    Liu, C, Bellec, GEF, Vogginger, B, Kappel, D, Partzsch, J, Neumärker, F, Höppner, S, Maass, W, Furber, SB, Legenstein, R & Mayr, CG 2018, 'Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype' Frontiers in neuroscience, Jg. 12, Nr. 840, 840. https://doi.org/10.3389/fnins.2018.00840
    Liu C, Bellec GEF, Vogginger B, Kappel D, Partzsch J, Neumärker F et al. Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype. Frontiers in neuroscience. 2018 Nov 19;12(840). 840. https://doi.org/10.3389/fnins.2018.00840
    Liu, Chen ; Bellec, Guillaume Emmanuel Fernand ; Vogginger, Bernhard ; Kappel, David ; Partzsch, Johannes ; Neumärker, Felix ; Höppner, Sebastian ; Maass, Wolfgang ; Furber, Steve B. ; Legenstein, Robert ; Mayr, Christian G. / Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype. in: Frontiers in neuroscience. 2018 ; Jahrgang 12, Nr. 840.
    @article{8a02314c19b84c669c1755478f0ae39b,
    title = "Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype",
    abstract = "The memory requirement of deep learning algorithms is considered incompatible with the memory restriction of energy-efficient hardware. A low memory footprint can be achieved by pruning obsolete connections or reducing the precision of connection strengths after the network has been trained. Yet, these techniques are not applicable to the case when neural networks have to be trained directly on hardware due to the hard memory constraints. Deep Rewiring (DEEP R) is a training algorithm which continuously rewires the network while preserving very sparse connectivity all along the training procedure.We apply DEEP R to a deep neural network implementation on a prototype chip of the 2ndgeneration SpiNNaker system. The local memory of a single core on this chip is limited to 64 KB and a deep network architecture is trained entirely within this constraint without the use of external memory. Throughout training, the proportion of active connections is limited to 1.3{\%}. On the handwritten digits dataset MNIST, this extremely sparse network achieves 96.6{\%} classification accuracy at convergence. Utilizing the multi-processor feature of the SpiNNaker system, we found very good scaling in terms of computation time, per-core memory consumption, and energy constraints. When compared to a X86 CPU implementation, neural network training on the SpiNNaker 2 prototype improvespower and energy consumption by two orders of magnitude.",
    keywords = "deep rewiring, pruning, sparsity, SpiNNaker, memory footprint, parallelism, energy efficient hardware",
    author = "Chen Liu and Bellec, {Guillaume Emmanuel Fernand} and Bernhard Vogginger and David Kappel and Johannes Partzsch and Felix Neum{\"a}rker and Sebastian H{\"o}ppner and Wolfgang Maass and Furber, {Steve B.} and Robert Legenstein and Mayr, {Christian G.}",
    year = "2018",
    month = "11",
    day = "19",
    doi = "10.3389/fnins.2018.00840",
    language = "English",
    volume = "12",
    journal = "Frontiers in neuroscience",
    issn = "1662-4548",
    publisher = "Frontiers Research Foundation",
    number = "840",

    }

    TY - JOUR

    T1 - Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype

    AU - Liu, Chen

    AU - Bellec, Guillaume Emmanuel Fernand

    AU - Vogginger, Bernhard

    AU - Kappel, David

    AU - Partzsch, Johannes

    AU - Neumärker, Felix

    AU - Höppner, Sebastian

    AU - Maass, Wolfgang

    AU - Furber, Steve B.

    AU - Legenstein, Robert

    AU - Mayr, Christian G.

    PY - 2018/11/19

    Y1 - 2018/11/19

    N2 - The memory requirement of deep learning algorithms is considered incompatible with the memory restriction of energy-efficient hardware. A low memory footprint can be achieved by pruning obsolete connections or reducing the precision of connection strengths after the network has been trained. Yet, these techniques are not applicable to the case when neural networks have to be trained directly on hardware due to the hard memory constraints. Deep Rewiring (DEEP R) is a training algorithm which continuously rewires the network while preserving very sparse connectivity all along the training procedure.We apply DEEP R to a deep neural network implementation on a prototype chip of the 2ndgeneration SpiNNaker system. The local memory of a single core on this chip is limited to 64 KB and a deep network architecture is trained entirely within this constraint without the use of external memory. Throughout training, the proportion of active connections is limited to 1.3%. On the handwritten digits dataset MNIST, this extremely sparse network achieves 96.6% classification accuracy at convergence. Utilizing the multi-processor feature of the SpiNNaker system, we found very good scaling in terms of computation time, per-core memory consumption, and energy constraints. When compared to a X86 CPU implementation, neural network training on the SpiNNaker 2 prototype improvespower and energy consumption by two orders of magnitude.

    AB - The memory requirement of deep learning algorithms is considered incompatible with the memory restriction of energy-efficient hardware. A low memory footprint can be achieved by pruning obsolete connections or reducing the precision of connection strengths after the network has been trained. Yet, these techniques are not applicable to the case when neural networks have to be trained directly on hardware due to the hard memory constraints. Deep Rewiring (DEEP R) is a training algorithm which continuously rewires the network while preserving very sparse connectivity all along the training procedure.We apply DEEP R to a deep neural network implementation on a prototype chip of the 2ndgeneration SpiNNaker system. The local memory of a single core on this chip is limited to 64 KB and a deep network architecture is trained entirely within this constraint without the use of external memory. Throughout training, the proportion of active connections is limited to 1.3%. On the handwritten digits dataset MNIST, this extremely sparse network achieves 96.6% classification accuracy at convergence. Utilizing the multi-processor feature of the SpiNNaker system, we found very good scaling in terms of computation time, per-core memory consumption, and energy constraints. When compared to a X86 CPU implementation, neural network training on the SpiNNaker 2 prototype improvespower and energy consumption by two orders of magnitude.

    KW - deep rewiring

    KW - pruning

    KW - sparsity

    KW - SpiNNaker

    KW - memory footprint

    KW - parallelism

    KW - energy efficient hardware

    U2 - 10.3389/fnins.2018.00840

    DO - 10.3389/fnins.2018.00840

    M3 - Article

    VL - 12

    JO - Frontiers in neuroscience

    JF - Frontiers in neuroscience

    SN - 1662-4548

    IS - 840

    M1 - 840

    ER -