Official codes: Self-Supervised Learning by Estimating Twin Class Distribution

Last update: Dec 15, 2022

Related tags

Overview

TWIST: Self-Supervised Learning by Estimating Twin Class Distributions

Codes and pretrained models for TWIST:

@article{wang2021self,
  title={Self-Supervised Learning by Estimating Twin Class Distributions},
  author={Wang, Feng and Kong, Tao and Zhang, Rufeng and Liu, Huaping and Li, Hang},
  journal={arXiv preprint arXiv:2110.07402},
  year={2021}
}

TWIST is a novel self-supervised representation learning method by classifying large-scale unlabeled datasets in an end-to-end way. We employ a siamese network terminated by a softmax operation to produce twin class distributions of two augmented images. Without supervision, we enforce the class distributions of different augmentations to be consistent. In the meantime, we regularize the class distributions to make them sharp and diverse. TWIST can naturally avoid the trivial solutions without specific designs such as asymmetric network, stop-gradient operation, or momentum encoder.

Models and Results

Main Models for Representation Learning

arch	params	epochs	linear	download
Model with multi-crop and self-labeling
ResNet-50	24M	850	75.5%	backbone only	full ckpt	args	log	eval logs
ResNet-50w2	94M	250	77.7%	backbone only	full ckpt	args	log	eval logs
DeiT-S	21M	300	75.6%	backbone only	full ckpt	args	log	eval logs
ViT-B	86M	300	77.3%	backbone only	full ckpt	args	log	eval logs
Model without multi-crop and self-labeling
ResNet-50	24M	800	72.6%	backbone only	full ckpt	args	log	eval logs

Model for unsupervised classification

arch	params	epochs	NMI	AMI	ARI	ACC	download
ResNet-50	24M	800	74.4	57.7	30.1	40.5	backbone only	full ckpt	args	log

Top-3 predictions for unsupervised classification

Semi-Supervised Results

arch	1% labels	10% labels	100% labels
resnet-50	61.5%	71.7%	78.4%
resnet-50w2	67.2%	75.3%	80.3%

Detection Results

Task	AP all	AP 50	AP 75
VOC07+12 detection	58.1	84.2	65.4
COCO detection	41.9	62.6	45.7
COCO instance segmentation	37.9	59.7	40.6

Single-node Training

ResNet-50 (requires 8 GPUs, Top-1 Linear 72.6%)

python3 -m torch.distributed.launch --nproc_per_node=8 --use_env train.py \
  --data-path ${DATAPATH} \
  --output_dir ${OUTPUT} \
  --aug barlow \
  --batch-size 256 \
  --dim 32768 \
  --epochs 800

Multi-node Training

ResNet-50 (requires 16 GPUs spliting over 2 nodes for multi-crop training, Top-1 Linear 75.5%)

python3 -m torch.distributed.launch --nproc_per_node=8 --use_env \
  --nnodes=${WORKER_NUM} \
  --node_rank=${MACHINE_ID} \
  --master_addr=${HOST} \
  --master_port=${PORT} train.py \
  --data-path ${DATAPATH} \
  --output_dir ${OUTPUT}

ResNet-50w2 (requires 32 GPUs spliting over 4 nodes for multi-crop training, Top-1 Linear 77.7%)

python3 -m torch.distributed.launch --nproc_per_node=8 --use_env \
  --nnodes=${WORKER_NUM} \
  --node_rank=${MACHINE_ID} \
  --master_addr=${HOST} \
  --master_port=${PORT} train.py \
  --data-path ${DATAPATH} \
  --output_dir ${OUTPUT} \
  --backbone 'resnet50w2' \
  --batch-size 60 \
  --bunch-size 240 \
  --epochs 250 \
  --mme_epochs 200

DeiT-S (requires 16 GPUs spliting over 2 nodes for multi-crop training, Top-1 Linear 75.6%)

python3 -m torch.distributed.launch --nproc_per_node=8 --use_env \
  --nnodes=${WORKER_NUM} \
  --node_rank=${MACHINE_ID} \
  --master_addr=${HOST} \
  --master_port=${PORT} train.py \
  --data-path ${DATAPATH} \
  --output_dir ${OUTPUT} \
  --backbone 'vit_s' \
  --batch-size 128 \
  --bunch-size 256 \
  --clip_norm 3.0 \
  --epochs 300 \
  --mme_epochs 300 \
  --lam1 -0.6 \
  --lam2 1.0 \
  --local_crops_number 6 \
  --lr 0.0005 \
  --momentum_start 0.996 \
  --momentum_end 1.0 \
  --optim admw \
  --use_momentum_encoder 1 \
  --weight_decay 0.06 \
  --weight_decay_end 0.06

ViT-B (requires 32 GPUs spliting over 4 nodes for multi-crop training, Top-1 Linear 77.3%)

python3 -m torch.distributed.launch --nproc_per_node=8 --use_env \
  --nnodes=${WORKER_NUM} \
  --node_rank=${MACHINE_ID} \
  --master_addr=${HOST} \
  --master_port=${PORT} train.py \
  --data-path ${DATAPATH} \
  --output_dir ${OUTPUT} \
  --backbone 'vit_b' \
  --batch-size 64 \
  --bunch-size 256 \
  --clip_norm 3.0 \
  --epochs 300 \
  --mme_epochs 300 \
  --lam1 -0.6 \
  --lam2 1.0 \
  --local_crops_number 6 \
  --lr 0.00075 \
  --momentum_start 0.996 \
  --momentum_end 1.0 \
  --optim admw \
  --use_momentum_encoder 1 \
  --weight_decay 0.06 \
  --weight_decay_end 0.06

Linear Classification

For ResNet-50

python3 evaluate.py \
  ${DATAPATH} \
  ${OUTPUT}/checkpoint.pth \
  --weight-decay 0 \
  --checkpoint-dir ${OUTPUT}/linear_multihead/ \
  --batch-size 1024 \
  --val_epoch 1 \
  --lr-classifier 0.2

For DeiT-S

python3 -m torch.distributed.launch --nproc_per_node=8 evaluate_vitlinear.py \
  --arch vit_s \
  --pretrained_weights ${OUTPUT}/checkpoint.pth \
  --lr 0.02 \
  --data_path ${DATAPATH} \
  --output_dir ${OUTPUT} \

For ViT-B

python3 -m torch.distributed.launch --nproc_per_node=8 evaluate_vitlinear.py \
  --arch vit_b \
  --pretrained_weights ${OUTPUT}/checkpoint.pth \
  --lr 0.0015 \
  --data_path ${DATAPATH} \
  --output_dir ${OUTPUT} \

Semi-supervised Learning

Command for training semi-supervised classification

1% Percent (61.5%)

python3 evaluate.py ${DATAPATH} ${MODELPATH} \
  --weights finetune \
  --lr-backbone 0.04 \
  --lr-classifier 0.2 \
  --train-percent 1 \
  --weight-decay 0 \
  --epochs 20 \
  --backbone 'resnet50'

10% Percent (71.7%)

python3 evaluate.py ${DATAPATH} ${MODELPATH} \
  --weights finetune \
  --lr-backbone 0.02 \
  --lr-classifier 0.2 \
  --train-percent 10 \
  --weight-decay 0 \
  --epochs 20 \
  --backbone 'resnet50'

100% Percent (78.4%)

python3 evaluate.py ${DATAPATH} ${MODELPATH} \
  --weights finetune \
  --lr-backbone 0.01 \
  --lr-classifier 0.2 \
  --train-percent 100 \
  --weight-decay 0 \
  --epochs 30 \
  --backbone 'resnet50'

Detection

Instruction

Install detectron2.

Convert a pre-trained MoCo model to detectron2's format:

python3 detection/convert-pretrain-to-detectron2.py ${MODELPATH} ${OUTPUTPKLPATH}

Put dataset under "detection/datasets" directory, following the directory structure requried by detectron2.

Training: VOC

cd detection/
python3 train_net.py \
  --config-file voc_fpn_1fc/pascal_voc_R_50_FPN_24k_infomin.yaml \
  --num-gpus 8 \
  MODEL.WEIGHTS ../${OUTPUTPKLPATH}

COCO

python3 train_net.py \
  --config-file infomin_configs/R_50_FPN_1x_infomin.yaml \
  --num-gpus 8 \
  MODEL.WEIGHTS ../${OUTPUTPKLPATH}

Official codes: Self-Supervised Learning by Estimating Twin Class Distribution

Related tags

Overview

TWIST: Self-Supervised Learning by Estimating Twin Class Distributions

Models and Results

Main Models for Representation Learning

Model for unsupervised classification

Top-3 predictions for unsupervised classification

Semi-Supervised Results

Detection Results

Single-node Training

Multi-node Training

Linear Classification

Semi-supervised Learning

Command for training semi-supervised classification

Detection

Instruction

Owner

Bytedance Inc.

In-place Parallel Super Scalar Samplesort (IPS⁴o)

A Kitti Road Segmentation model implemented in tensorflow.

Hierarchical Motion Encoder-Decoder Network for Trajectory Forecasting (HMNet)

MMGeneration is a powerful toolkit for generative models, based on PyTorch and MMCV.

DL course co-developed by YSDA, HSE and Skoltech

UT-Sarulab MOS prediction system using SSL models

CTF challenges from redpwnCTF 2021

Public scripts, services, and configuration for running a smart home K3S network cluster

g9.py - Torch interactive graphics

Self-Learning - Books Papers, Courses & more I have to learn soon

PyTorch3D is FAIR's library of reusable components for deep learning with 3D data

Hypernetwork-Ensemble Learning of Segmentation Probability for Medical Image Segmentation with Ambiguous Labels

performing moving objects segmentation using image processing techniques with opencv and numpy

Repo for EchoVPR: Echo State Networks for Visual Place Recognition

This is the official PyTorch implementation of the paper "TransFG: A Transformer Architecture for Fine-grained Recognition" (Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang, Yutong Bai, Changhu Wang, Alan Yuille).

Outlier Exposure with Confidence Control for Out-of-Distribution Detection

A new benchmark for Icon Question Answering (IconQA) and a large-scale icon dataset Icon645.

Code for "ShineOn: Illuminating Design Choices for Practical Video-based Virtual Clothing Try-on", accepted at WACV 2021 Generation of Human Behavior Workshop.

PyTorch implementation of SCAFFOLD (Stochastic Controlled Averaging for Federated Learning, ICML 2020).

[NeurIPS 2021] Introspective Distillation for Robust Question Answering