Implementation for paper "Towards the Generalization of Contrastive Self-Supervised Learning"

Last update: Nov 30, 2022

Overview

Contrastive Self-Supervised Learning on CIFAR-10

Paper

"Towards the Generalization of Contrastive Self-Supervised Learning", Weiran Huang, Mingyang Yi and Xuyang Zhao, arXiv:2111.00743, 2021.

The most critical argument we made in our paper is that the quality of data augmentation exhibits great impact on the quality of contrastive-learned encoder. The data augmentation with sharper intra-class concentration enables the model to have better generalization on downstream tasks. We verify it through a variety of experiments in this repository.

Supported methods

SimCLR
Barlow Twins
MoCo
SimSiam

Installation

pip install -r requirement.txt

Dependencies

torch==1.4.0
torchvision==0.5.0
pytorch-lightning==1.3.8
lightly==1.0.8 (important!)

Evaluation

KNN evaluation protocol. Code from here.

Results

ResNet-18 trained by SimCLR and Barlow Twins over various data augmentation combinations.

Example: python main.py --model=twins --epoch=800 --batch=512 --round=3 --augs=abcde

(a)	(b)	(c)	(d)	(e)	SimCLR	Barlow Twins
✓	✓	✓	✓	✓	89.92 ± 0.05	83.93 ± 0.57
✓	✓	✓	✓	×	88.41 ± 0.11	83.37 ± 0.43
✓	✓	✓	×	×	83.62 ± 0.19	73.70 ± 0.99
✓	✓	×	×	×	62.91 ± 0.25	49.56 ± 0.11
✓	×	×	×	×	62.37 ± 0.09	48.54 ± 0.29

Augmentation operations include:

(a) random cropping with a scaling factor chosen in [0.08, 1.0];

(b) random Gaussian blur with a probability 0.5;

(d) color distortion with a probability of 0.8 and with strength of [0.4, 0.4, 0.4, 0.1];

(e) random horizontal flipping with a probability of 0.5.

ResNet18 trained by SimCLR and Barlow Twins over various color distortion strengths.

Example: python main.py --model=simclr --epoch=800 --batch=512 --round=3 --augs=color --strength=1

Color Distortion Strength	SimCLR	Barlow Twins
1/8	73.60 ± 0.11	61.13 ± 2.81
1/4	76.25 ± 0.16	68.30 ± 0.15
1/2	78.49 ± 0.09	72.76 ± 1.50
1	82.64 ± 0.57	78.79 ± 0.54

Acknowledgement

This code is based on:

Citation

If you find our work useful in your research, please consider citing:

@misc{huang2021generalization,
      title={Towards the Generalization of Contrastive Self-Supervised Learning}, 
      author={Weiran Huang and Mingyang Yi and Xuyang Zhao},
      year={2021},
      eprint={2111.00743},
      archivePrefix={arXiv}
}

You might also like...

Implementation of Nyström Self-attention, from the paper Nyströmformer

Nyström Attention Implementation of Nyström Self-attention, from the paper Nyströmformer. Yannic Kilcher video Install $ pip install nystrom-attention

95 Jan 2, 2023

Implementation of SETR model, Original paper: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.

SETR - Pytorch Since the original paper (Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.) has no official

112 Dec 16, 2022

Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

CIPS -- Official Pytorch Implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis Requirements pip install -r requi

Multimodal Lab @ Samsung AI Center Moscow

201 Dec 21, 2022

Official pytorch implementation of paper

Implementation for paper "Towards the Generalization of Contrastive Self-Supervised Learning"

Related tags

Overview

Contrastive Self-Supervised Learning on CIFAR-10

Paper

Supported methods

Installation

Dependencies

Evaluation

Results

ResNet-18 trained by SimCLR and Barlow Twins over various data augmentation combinations.

ResNet18 trained by SimCLR and Barlow Twins over various color distortion strengths.

Acknowledgement

Citation

You might also like...

Implementation of Nyström Self-attention, from the paper Nyströmformer

Implementation of SETR model, Original paper: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.

Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

Official pytorch implementation of paper "Image-to-image Translation via Hierarchical Style Disentanglement".

PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 2021

Implementation of Barlow Twins paper

Official pytorch implementation of paper "Inception Convolution with Efficient Dilation Search" (CVPR 2021 Oral).

Official implementation of our paper "LLA: Loss-aware Label Assignment for Dense Pedestrian Detection" in Pytorch.

Functional TensorFlow Implementation of Singular Value Decomposition for paper Fast Graph Learning

Releases(v0.2.1)

v0.2.1(Jun 16, 2022)

v0.2.0(Jun 4, 2022)

v0.1.0(Jun 4, 2022)

Owner

Weiran Huang

Rendering Point Clouds with Compute Shaders

A keras implementation of ENet (abandoned for the foreseeable future)

RetinaNet-PyTorch - A RetinaNet Pytorch Implementation on remote sensing images and has the similar mAP result with RetinaNet in MMdetection

pip install python-office

A Keras implementation of CapsNet in the paper: Sara Sabour, Nicholas Frosst, Geoffrey E Hinton. Dynamic Routing Between Capsules

Vit-ImageClassification - Pytorch ViT for Image classification on the CIFAR10 dataset

Fine-tuning StyleGAN2 for Cartoon Face Generation

(ImageNet pretrained models) The official pytorch implemention of the TPAMI paper "Res2Net: A New Multi-scale Backbone Architecture"

Whisper is a file-based time-series database format for Graphite.

Cluttered MNIST Dataset

EfficientDet (Scalable and Efficient Object Detection) implementation in Keras and Tensorflow

PyTorch Implementation of Daft-Exprt: Robust Prosody Transfer Across Speakers for Expressive Speech Synthesis

SSD: A Unified Framework for Self-Supervised Outlier Detection [ICLR 2021]

Official implementation of our CVPR2021 paper "OTA: Optimal Transport Assignment for Object Detection" in Pytorch.

Regulatory Instruments for Fair Personalized Pricing.

True per-item rarity for Loot

A robust camera and Lidar fusion based velocity estimator to undistort the pointcloud.

The aim of this project is to build an AI bot that can play the Wordle game, or more generally Squabble

The authors' official PyTorch SigWGAN implementation

This is the pytorch re-implementation of the IterNorm