Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle

Rana Ali Amjad, Bernhard Geiger

Publikation: Beitrag in einer FachzeitschriftArtikelBegutachtung

Abstract

In this theory paper, we investigate training deep neural networks (DNNs) for classification via minimizing the information bottleneck (IB) functional. We show that the resulting optimization problem suffers from two severe issues: First, for deterministic DNNs, either the IB functional is infinite for almost all values of network parameters, making the optimization problem ill-posed, or it is piecewise constant, hence not admitting gradient-based optimization methods. Second, the invariance of the IB functional under bijections prevents it from capturing properties of the learned representation that are desirable for classification, such as robustness and simplicity. We argue that these issues are partly resolved for stochastic DNNs, DNNs that include a (hard or soft) decision rule, or by replacing the IB functional with related, but more well-behaved cost functions. We conclude that recent successes reported about training DNNs using the IB framework must be attributed to such solutions. As a side effect, our results indicate limitations of the IB framework for the analysis of DNNs. We also note that rather than trying to repair the inherent problems in the IB functional, a better approach may be to design regularizers on latent representation enforcing the desired properties directly.

Originalspracheenglisch
Aufsatznummer8680020
Seiten (von - bis)2225-2239
Seitenumfang15
FachzeitschriftIEEE Transactions on Pattern Analysis and Machine Intelligence
Jahrgang42
Ausgabenummer9
DOIs
PublikationsstatusVeröffentlicht - 1 Sept. 2020

ASJC Scopus subject areas

  • Software
  • Artificial intelligence
  • Angewandte Mathematik
  • Maschinelles Sehen und Mustererkennung
  • Theoretische Informatik und Mathematik

Fingerprint

Untersuchen Sie die Forschungsthemen von „Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle“. Zusammen bilden sie einen einzigartigen Fingerprint.

Dieses zitieren