A Comparison of Variational Bounds for the Information Bottleneck Functional

Bernhard Geiger, Ian Fischer

Research output: Contribution to journalArticlepeer-review

Abstract

In this short note, we relate the variational bounds proposed in Alemi et al. (2017) and Fischer (2020) for the information bottleneck (IB) and the conditional entropy bottleneck (CEB) functional, respectively. Although the two functionals were shown to be equivalent, it was empirically observed that optimizing bounds on the CEB functional achieves better generalization performance and adversarial robustness than optimizing those on the IB functional. This work tries to shed light on this issue by showing that, in the most general setting, no ordering can be established between these variational bounds, while such an ordering can be enforced by restricting the feasible sets over which the optimizations take place. The absence of such an ordering in the general setup suggests that the variational bound on the CEB functional is either more amenable to optimization or a relevant cost function for optimization in its own regard, i.e., without justification from the IB or CEB functionals.

Original languageEnglish
Article number1229
Pages (from-to)1-12
Number of pages12
JournalEntropy
Volume22
Issue number11
DOIs
Publication statusPublished - Nov 2020

Fingerprint

Dive into the research topics of 'A Comparison of Variational Bounds for the Information Bottleneck Functional'. Together they form a unique fingerprint.

Cite this