[NIPS 2021] UOTA: Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration.

Last update: Jun 29, 2022

Related tags

Deep Learning uota

Overview

UOTA: Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration

This repository is the official PyTorch implementation of UOTA (Unsupervised OuTlier Arbitration).

0 Requirements

Python 3.6
PyTorch install = 1.6.0
torchvision install = 0.7.0
CUDA 10.1
Apex with CUDA extension
Other dependencies: opencv-python, scipy, pandas, numpy

1 Pretraining

We release a demo to pretrain ResNet50 on ImageNet1K with SwAV+UOTA pretrained models.

1.1 SwAV+UOTA pretrain

To train SwAV+UOTA on a single node with 4 gpus for 200 epochs, run:

DATASET_PATH="path/to/ImageNet1K/train"
EXPERIMENT_PATH="path/to/experiment"

python -m torch.distributed.launch --nproc_per_node=4 main_uota.py \
--data_path ${DATASET_PATH} \
--nmb_crops 2 6 \
--size_crops 224 96 \
--min_scale_crops 0.14 0.05 \
--max_scale_crops 1. 0.14 \
--crops_for_assign 0 1 \
--use_pil_blur true \
--epochs 200 \
--warmup_epochs 0 \
--batch_size 64 \
--base_lr 0.6 \
--final_lr 0.0006 \
--uota_tau 350. \
--epoch_uota_starts 100 \
--wd 0.000001 \
--use_fp16 true \
--dist_url "tcp://localhost:40000" \
--arch uota_r50 \
--sync_bn pytorch \
--dump_path ${EXPERIMENT_PATH}

2 Linear Evaluation

To train a linear classifier on frozen features out of deep network pretrained via various self-supervised pretraining methods, run:

DATASET_PATH="path/to/ImageNet1K"
EXPERIMENT_PATH="path/to/experiment"
LINCLS_PATH="path/to/lincls"

python -m torch.distributed.launch --nproc_per_node=4 eval_linear.py \
--data_path ${DATASET_PATH} \
--arch resnet50 \
--lr 1.2 \
--dump_path ${LINCLS_PATH} \
--pretrained ${EXPERIMENT_PATH}/swav_uota_r50_e200_pretrained.pth \
--batch_size 64 \
--num_classes 100 \

3 Results

To compare with SwAV fairly, we provide a SwAV+UOTA model with ResNet-50 architecture pretrained on ImageNet1K for 200 epochs, and release the pretrained model and the linear classier.

method	epochs	batch-size	multi-crop	ImageNet1K top-1 acc.	pretrained model	linear classifier
SwAV	200	256	2x224 + 6x96	72.7	/	/
SwAV + UOTA	200	256	2x224 + 6x96	73.5	pretrained	linear

4 Citation

@InProceedings{wang2021NeurIPS,
  title={Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration},
  author={Wang, Yu and Lin, Jingyang and Zou, Jingjing and Pan, Yingwei and Yao, Ting and Mei, Tao},
  booktitle={NeurIPS},
  year={2021},
}

You might also like...

PyTorch implementation of spectral graph ConvNets, NIPS’16

Graph ConvNets in PyTorch October 15, 2017 Xavier Bresson http://www.ntu.edu.sg/home/xbresson https://github.com/xbresson https://twitter.com/xbresson

287 Jan 4, 2023

PyTorch implementation of the Value Iteration Networks (VIN) (NIPS '16 best paper)

Value Iteration Networks in PyTorch Tamar, A., Wu, Y., Thomas, G., Levine, S., and Abbeel, P. Value Iteration Networks. Neural Information Processing

75 Nov 24, 2022

Pytorch implementation of Value Iteration Networks (NIPS 2016 best paper)

VIN: Value Iteration Networks A quick thank you A few others have released amazing related work which helped inspire and improve my own implementation

297 Dec 26, 2022

pytorch implementation of "Contrastive Multiview Coding", "Momentum Contrast for Unsupervised Visual Representation Learning", and "Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination"

Unofficial implementation: MoCo: Momentum Contrast for Unsupervised Visual Representation Learning (Paper) InsDis: Unsupervised Feature Learning via N

16 Nov 4, 2020

The official implementation of CVPR 2021 Paper: Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation.

Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation This repository is the official implementation of CVPR 2021 paper:

9 Nov 14, 2022

(JMLR'19) A Python Toolbox for Scalable Outlier Detection (Anomaly Detection)

[NIPS 2021] UOTA: Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration.

Related tags

Overview

UOTA: Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration

0 Requirements

1 Pretraining

1.1 SwAV+UOTA pretrain

2 Linear Evaluation

3 Results

4 Citation

You might also like...

PyTorch implementation of spectral graph ConvNets, NIPS’16

PyTorch implementation of the Value Iteration Networks (VIN) (NIPS '16 best paper)

Pytorch implementation of Value Iteration Networks (NIPS 2016 best paper)

pytorch implementation of "Contrastive Multiview Coding", "Momentum Contrast for Unsupervised Visual Representation Learning", and "Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination"

The official implementation of CVPR 2021 Paper: Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation.

(JMLR'19) A Python Toolbox for Scalable Outlier Detection (Anomaly Detection)

Streaming Anomaly Detection Framework in Python (Outlier Detection for Streaming Data)

A gesture recognition system powered by OpenPose, k-nearest neighbours, and local outlier factor.

Outlier Exposure with Confidence Control for Out-of-Distribution Detection

Releases(v1.0.0)

v1.0.0(Oct 26, 2021)

Owner

A quick recipe to learn all about Transformers

The world's largest toxicity dataset.

This repository contains the code used to quantitatively evaluate counterfactual examples in the associated paper.

Keras implementation of PersonLab for Multi-Person Pose Estimation and Instance Segmentation.

Pytorch ImageNet1k Loader with Bounding Boxes.

A Fast and Accurate One-Stage Approach to Visual Grounding, ICCV 2019 (Oral)

Convert scikit-learn models to PyTorch modules

Pytorch implementation of Straight Sampling Network For Point Cloud Learning (ICIP2021).

An implementation of the 1. Parallel, 2. Streaming, 3. Randomized SVD using MPI4Py

Code for Temporally Abstract Partial Models

How Do Adam and Training Strategies Help BNNs Optimization? In ICML 2021.

Mix3D: Out-of-Context Data Augmentation for 3D Scenes (3DV 2021)

A tool to estimate time varying instantaneous reproduction number during epidemics

A Python 3 package for state-of-the-art statistical dimension reduction methods

Create UIs for prototyping your machine learning model in 3 minutes

Convert dog pictures into various painting styles. Try LimnPet

SymmetryNet: Learning to Predict Reflectional and Rotational Symmetries of 3D Shapes from Single-View RGB-D Images

Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners

Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning, CVPR 2021

End-to-end beat and downbeat tracking in the time domain.