The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks

Rahim Entezari; Hanie Sedghi; Olga Saukh; Behnam Neyshabur

The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks

Rahim Entezari, Hanie Sedghi, Olga Saukh, Behnam Neyshabur

Institute of Technical Informatics (4480)

Research output: Contribution to conference › Paper › peer-review

Abstract

In this paper, we conjecture that if the permutation invariance of neural networks is taken into account, SGD solutions will likely have no barrier in the linear interpolation between them. Although it is a bold conjecture, we show how extensive empirical attempts fall short of refuting it. We further provide a preliminary theoretical result to support our conjecture. Our conjecture has implications for lottery ticket hypothesis, distributed training, and ensemble methods.

Original language	English
Publication status	Accepted/In press - 25 Jan 2022
Event	10th International Conference on Learning Representations: ICLR 2022 - online, Virtuell Duration: 25 Apr 2022 → 29 Apr 2022 https://iclr.cc/

Conference

Conference	10th International Conference on Learning Representations
Abbreviated title	ICLR 2022
City	Virtuell
Period	25/04/22 → 29/04/22
Internet address	https://iclr.cc/

Keywords

cs.LG

Access to Document

2110.06296v1

Cite this

@conference{e6cf42a07902476a8b49dbd18e949f2f,

title = "The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks",

abstract = " In this paper, we conjecture that if the permutation invariance of neural networks is taken into account, SGD solutions will likely have no barrier in the linear interpolation between them. Although it is a bold conjecture, we show how extensive empirical attempts fall short of refuting it. We further provide a preliminary theoretical result to support our conjecture. Our conjecture has implications for lottery ticket hypothesis, distributed training, and ensemble methods. ",

keywords = "cs.LG",

author = "Rahim Entezari and Hanie Sedghi and Olga Saukh and Behnam Neyshabur",

year = "2022",

month = jan,

day = "25",

language = "English",

note = "10th International Conference on Learning Representations : ICLR 2022, ICLR 2022 ; Conference date: 25-04-2022 Through 29-04-2022",

url = "https://iclr.cc/",

}

TY - CONF

T1 - The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks

AU - Entezari, Rahim

AU - Sedghi, Hanie

AU - Saukh, Olga

AU - Neyshabur, Behnam

PY - 2022/1/25

Y1 - 2022/1/25

N2 - In this paper, we conjecture that if the permutation invariance of neural networks is taken into account, SGD solutions will likely have no barrier in the linear interpolation between them. Although it is a bold conjecture, we show how extensive empirical attempts fall short of refuting it. We further provide a preliminary theoretical result to support our conjecture. Our conjecture has implications for lottery ticket hypothesis, distributed training, and ensemble methods.

AB - In this paper, we conjecture that if the permutation invariance of neural networks is taken into account, SGD solutions will likely have no barrier in the linear interpolation between them. Although it is a bold conjecture, we show how extensive empirical attempts fall short of refuting it. We further provide a preliminary theoretical result to support our conjecture. Our conjecture has implications for lottery ticket hypothesis, distributed training, and ensemble methods.

KW - cs.LG

M3 - Paper

T2 - 10th International Conference on Learning Representations

Y2 - 25 April 2022 through 29 April 2022

ER -

The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks

Abstract

Conference

Keywords

Access to Document

Fingerprint

Cite this