Studying the impact of magnitude pruning on contrastive learning methods

Francesco Corti*, Rahim Entezari*, Sara Hooker, Davide Bacciu, Olga Saukh

*Corresponding author for this work

Research output: Contribution to conferencePosterpeer-review

Abstract

We study the impact of different pruning techniques on the representation learned by deep neural networks trained with contrastive loss functions. Our work finds that at high sparsity levels, contrastive learning results in a higher number of misclassified examples relative to models trained with traditional cross-entropy loss. To understand this pronounced difference, we use metrics such as the number of PIEs (Hooker et al., 2019), Q-Score (Kalibhat et al., 2022), and PD-Score (Baldock et al., 2021) to measure the impact of pruning on the learned representation quality. Our analysis suggests the schedule of the pruning method implementation matters. We find that the negative impact of sparsity on the quality of the learned representation is the highest when pruning is introduced early on in the training phase.
Original languageEnglish
Number of pages10
Publication statusPublished - 1 Jul 2022
Event39th International Conference on Machine
Learning: ICML 2022
- Baltimore, United States
Duration: 17 Jul 202223 Jul 2022

Conference

Conference39th International Conference on Machine
Learning
Abbreviated titleICML 2022
Country/TerritoryUnited States
CityBaltimore
Period17/07/2223/07/22

Fields of Expertise

  • Information, Communication & Computing

Fingerprint

Dive into the research topics of 'Studying the impact of magnitude pruning on contrastive learning methods'. Together they form a unique fingerprint.

Cite this