Deep Rewiring: Training very sparse deep networks

Guillaume Bellec, David Kappel, Wolfgang Maass, Robert Legenstein

Research output: Contribution to journalArticleResearch

Abstract

Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them. But also generic hardware and software implementations of deep learning run more efficiently on sparse networks. Several methods exist for pruning connections of a neural network after it was trained without connectivity constraints. We present an algorithm, DEEP R, that enables us to train directly a sparsely connected neural network. DEEP R automatically rewires the network during supervised training so that connections are there where they are most needed for the task, while its total number is all the time strictly bounded. We demonstrate that DEEP R can be used to train very sparse feedforward and recurrent neural networks on standard benchmark tasks with just a minor loss in performance. DEEP R is based on a rigorous theoretical foundation that views rewiring as stochastic sampling of network configurations from a posterior.
Original languageEnglish
JournalarXiv.org e-Print archive
Publication statusPublished - 14 Nov 2017

Fingerprint

Neural networks
Hardware
Recurrent neural networks
Feedforward neural networks
Sampling
Deep learning

Keywords

  • cs.NE
  • cs.AI
  • cs.DC
  • cs.LG
  • stat.ML

Cite this

Deep Rewiring : Training very sparse deep networks. / Bellec, Guillaume; Kappel, David; Maass, Wolfgang; Legenstein, Robert.

In: arXiv.org e-Print archive, 14.11.2017.

Research output: Contribution to journalArticleResearch

@article{97aa7d8d2e4b44c8982025f12542014a,
title = "Deep Rewiring: Training very sparse deep networks",
abstract = "Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them. But also generic hardware and software implementations of deep learning run more efficiently on sparse networks. Several methods exist for pruning connections of a neural network after it was trained without connectivity constraints. We present an algorithm, DEEP R, that enables us to train directly a sparsely connected neural network. DEEP R automatically rewires the network during supervised training so that connections are there where they are most needed for the task, while its total number is all the time strictly bounded. We demonstrate that DEEP R can be used to train very sparse feedforward and recurrent neural networks on standard benchmark tasks with just a minor loss in performance. DEEP R is based on a rigorous theoretical foundation that views rewiring as stochastic sampling of network configurations from a posterior.",
keywords = "cs.NE, cs.AI, cs.DC, cs.LG, stat.ML",
author = "Guillaume Bellec and David Kappel and Wolfgang Maass and Robert Legenstein",
note = "10 pages (11 with references, 21 with appendix), 4 Figures in the main text, submitted as a conference paper at ICLR 2018",
year = "2017",
month = "11",
day = "14",
language = "English",
journal = "arXiv.org e-Print archive",
publisher = "Cornell University Library",

}

TY - JOUR

T1 - Deep Rewiring

T2 - Training very sparse deep networks

AU - Bellec, Guillaume

AU - Kappel, David

AU - Maass, Wolfgang

AU - Legenstein, Robert

N1 - 10 pages (11 with references, 21 with appendix), 4 Figures in the main text, submitted as a conference paper at ICLR 2018

PY - 2017/11/14

Y1 - 2017/11/14

N2 - Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them. But also generic hardware and software implementations of deep learning run more efficiently on sparse networks. Several methods exist for pruning connections of a neural network after it was trained without connectivity constraints. We present an algorithm, DEEP R, that enables us to train directly a sparsely connected neural network. DEEP R automatically rewires the network during supervised training so that connections are there where they are most needed for the task, while its total number is all the time strictly bounded. We demonstrate that DEEP R can be used to train very sparse feedforward and recurrent neural networks on standard benchmark tasks with just a minor loss in performance. DEEP R is based on a rigorous theoretical foundation that views rewiring as stochastic sampling of network configurations from a posterior.

AB - Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them. But also generic hardware and software implementations of deep learning run more efficiently on sparse networks. Several methods exist for pruning connections of a neural network after it was trained without connectivity constraints. We present an algorithm, DEEP R, that enables us to train directly a sparsely connected neural network. DEEP R automatically rewires the network during supervised training so that connections are there where they are most needed for the task, while its total number is all the time strictly bounded. We demonstrate that DEEP R can be used to train very sparse feedforward and recurrent neural networks on standard benchmark tasks with just a minor loss in performance. DEEP R is based on a rigorous theoretical foundation that views rewiring as stochastic sampling of network configurations from a posterior.

KW - cs.NE

KW - cs.AI

KW - cs.DC

KW - cs.LG

KW - stat.ML

M3 - Article

JO - arXiv.org e-Print archive

JF - arXiv.org e-Print archive

ER -