Improving Transferability of Representations via Augmentation-Aware Self-Supervision

Last update: Sep 16, 2022

Related tags

Overview

Improving Transferability of Representations via Augmentation-Aware Self-Supervision

Accepted to NeurIPS 2021

TL;DR: Learning augmentation-aware information by predicting the difference between two augmented samples improves the transferability of representations.

Dependencies

conda create -n AugSelf python=3.8 pytorch=1.7.1 torchvision=0.8.2 cudatoolkit=10.1 ignite -c pytorch
conda activate AugSelf
pip install scipy tensorboard kornia==0.4.1 sklearn

Checkpoints

We provide ImageNet100-pretrained models in this Dropbox link.

Pretraining

We here provide SimSiam+AugSelf pretraining scripts. For training the baseline (i.e., no AugSelf), remove --ss-crop and --ss-color options. For using other frameworks like SimCLR, use the --framework option.

STL-10

CUDA_VISIBLE_DEVICES=0 python pretrain.py \
    --logdir ./logs/stl10/simsiam/aug_self \
    --framework simsiam \
    --dataset stl10 \
    --datadir DATADIR \
    --model resnet18 \
    --batch-size 256 \
    --max-epochs 200 \
    --ss-color 1.0 --ss-crop 1.0

ImageNet100

python pretrain.py \
    --logdir ./logs/imagenet100/simsiam/aug_self \
    --framework simsiam \
    --dataset imagenet100 \
    --datadir DATADIR \
    --batch-size 256 \
    --max-epochs 500 \
    --model resnet50 \
    --base-lr 0.05 --wd 1e-4 \
    --ckpt-freq 50 --eval-freq 50 \
    --ss-crop 0.5 --ss-color 0.5 \
    --num-workers 16 --distributed

Evaluation

Our main evaluation setups are linear evaluation on fine-grained classification datasets (Table 1) and few-shot benchmarks (Table 2).

linear evaluation

CUDA_VISIBLE_DEVICES=0 python transfer_linear_eval.py \
    --pretrain-data imagenet100 \
    --ckpt CKPT \
    --model resnet50 \
    --dataset cifar10 \
    --datadir DATADIR \
    --metric top1

few-shot

CUDA_VISIBLE_DEVICES=0 python transfer_few_shot.py \
    --pretrain-data imagenet100 \
    --ckpt CKPT \
    --model resnet50 \
    --dataset cub200 \
    --datadir DATADIR

Improving Transferability of Representations via Augmentation-Aware Self-Supervision

Related tags

Overview

Improving Transferability of Representations via Augmentation-Aware Self-Supervision

Dependencies

Checkpoints

Pretraining

STL-10

ImageNet100

Evaluation

linear evaluation

few-shot

Owner

hankook

Easily Process a Batch of Cox Models

Implementation of our recent paper, WOOD: Wasserstein-based Out-of-Distribution Detection.

Cockpit is a visual and statistical debugger specifically designed for deep learning.

An official source code for "Augmentation-Free Self-Supervised Learning on Graphs"

Storchastic is a PyTorch library for stochastic gradient estimation in Deep Learning

An implementation of Deep Forest 2021.2.1.

An implementation of the paper "A Neural Algorithm of Artistic Style"

Codes for our paper The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders published to EMNLP 2021.

LocUNet is a deep learning method to localize a UE based solely on the reported signal strengths from a set of BSs.

naked is a Python tool which allows you to strip a model and only keep what matters for making predictions.

ObsPy: A Python Toolbox for seismology/seismological observatories.

MAT: Mask-Aware Transformer for Large Hole Image Inpainting

Affine / perspective transformation in Pose Estimation with Tensorflow 2

Train an imgs.ai model on your own dataset

tmm_fast is a lightweight package to speed up optical planar multilayer thin-film device computation.

EfficientDet (Scalable and Efficient Object Detection) implementation in Keras and Tensorflow

A Data Annotation Tool for Semantic Segmentation, Object Detection and Lane Line Detection.(In Development Stage)

PyTorch implementation of our Adam-NSCL algorithm from our CVPR2021 (oral) paper "Training Networks in Null Space for Continual Learning"

Differential Privacy for Heterogeneous Federated Learning : Utility & Privacy tradeoffs

Code Release for ICCV 2021 (oral), "AdaFit: Rethinking Learning-based Normal Estimation on Point Clouds"