A Comparison of Variational Bounds for the Information Bottleneck Functional

Bernhard Geiger, Ian Fischer

Publikation: Beitrag in einer FachzeitschriftArtikel

Abstract

In this short note, we relate the variational bounds proposed in Alemi et al. (2017) and Fischer (2020) for the information bottleneck (IB) and the conditional entropy bottleneck (CEB) functional, respectively. Although the two functionals were shown to be equivalent, it was empirically observed that optimizing bounds on the CEB functional achieves better generalization performance and adversarial robustness than optimizing those on the IB functional. This work tries to shed light on this issue by showing that, in the most general setting, no ordering can be established between these variational bounds, while such an ordering can be enforced by restricting the feasible sets over which the optimizations take place. The absence of such an ordering in the general setup suggests that the variational bound on the CEB functional is either more amenable to optimization or a relevant cost function for optimization in its own regard, i.e., without justification from the IB or CEB functionals.

Originalspracheenglisch
Aufsatznummer1229
Seiten (von - bis)1-12
Seitenumfang12
FachzeitschriftEntropy
Jahrgang22
Ausgabenummer11
DOIs
PublikationsstatusVeröffentlicht - Nov 2020

Fingerprint Untersuchen Sie die Forschungsthemen von „A Comparison of Variational Bounds for the Information Bottleneck Functional“. Zusammen bilden sie einen einzigartigen Fingerprint.

Dieses zitieren