Skip to content

[CVPR 2022 Oral] Rethinking Minimal Sufficient Representation in Contrastive Learning

Notifications You must be signed in to change notification settings

Haoqing-Wang/InfoCL

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

10 Commits
 
 
 
 
 
 

Repository files navigation

Rethinking Minimal Sufficient Representation in Contrastive Learning

PyTorch implementation of
Rethinking Minimal Sufficient Representation in Contrastive Learning
Haoqing Wang, Xun Guo, Zhi-hong Deng, Yan Lu

CVPR 2022 Oral (Top 4.2%)

Abstract

Contrastive learning between different views of the data achieves outstanding success in the field of self-supervised representation learning and the learned representations are useful in broad downstream tasks. Since all supervision information for one view comes from the other view, contrastive learning approximately obtains the minimal sufficient representation which contains the shared information and eliminates the non-shared information between views. Considering the diversity of the downstream tasks, it cannot be guaranteed that all task-relevant information is shared between views. Therefore, we assume the non-shared task-relevant information cannot be ignored and theoretically prove that the minimal sufficient representation in contrastive learning is not sufficient for the downstream tasks, which causes performance degradation. This reveals a new problem that the contrastive learning models have the risk of over-fitting to the shared information between views. To alleviate this problem, we propose to increase the mutual information between the representation and input as regularization to approximately introduce more task-relevant information, since we cannot utilize any downstream task information during training. Extensive experiments verify the rationality of our analysis and the effectiveness of our method. It significantly improves the performance of several classic contrastive learning models in downstream tasks.

Citation

If you use this code for your research, please cite our paper:

@inproceedings{wang2022rethinking,
  title={Rethinking Minimal Sufficient Representation in Contrastive Learning},
  author={Wang, Haoqing and Guo, Xun and Deng, Zhi-Hong and Lu, Yan},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={16041--16050},
  year={2022}
}

Note

  • This code is built upon the implementation from moco and CLAE.
  • The dataset, model, and code are for non-commercial research purposes only.

About

[CVPR 2022 Oral] Rethinking Minimal Sufficient Representation in Contrastive Learning

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published