REPAIR: REnormalizing Permuted Activations for Interpolation Repair

Keller Jordan*, Hanie Sedghi, Olga Saukh, Rahim Entezari, Behnam Neyshabur

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference paperpeer-review


In this paper we look into the conjecture of Entezari et al. (2021) which states that if the permutation invariance of neural networks is taken into account, then there is likely no loss barrier to the linear interpolation between SGD solutions. First, we observe that neuron alignment methods alone are insufficient to establish low-barrier linear connectivity between SGD solutions due to a phenomenon we call variance collapse: interpolated deep networks suffer a collapse in the variance of their activations, causing poor performance. Next, we propose REPAIR (REnormalizing Permuted Activations for Interpolation Repair) which mitigates variance collapse by rescaling the preactivations of such interpolated networks. We explore the interaction between our method and the choice of normalization layer, network width, and depth, and demonstrate that using REPAIR on top of neuron alignment methods leads to 60%-100% relative barrier reduction across a wide variety of architecture families and tasks. In particular, we report a 74% barrier reduction for ResNet50 on ImageNet and 90% barrier reduction for ResNet18 on CIFAR10
Original languageEnglish
Title of host publicationInternational Conference On Representation Learning
Publication statusAccepted/In press - 20 Jan 2023


  • deep learning
  • permutations
  • loss landscape
  • generalization


Dive into the research topics of 'REPAIR: REnormalizing Permuted Activations for Interpolation Repair'. Together they form a unique fingerprint.

Cite this