[ICML'21] Estimate the accuracy of the classifier in various environments through self-supervision

Overview

What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?

[Paper] [ICML'21 Project]

PyTorch Implementation

This repository contains:

  • the PyTorch implementation of AutoEavl.
  • the example on CIFAR-10 setup (use imgaug)
  • linear regression

Please follow the instruction below to install it and run the experiment demo.

Prerequisites

  • Linux (tested on Ubuntu 16.04LTS)
  • NVIDIA GPU + CUDA CuDNN (tested on GTX 2080 Ti)
  • CIFAR-10 (download and unzip to PROJECT_DIR/data/)
  • CIFAR10.1 (download and unzip to PROJECT_DIR/data/CIFAR-10.1)
  • Please use PyTorch1.5 to avoid compilation errors (other versions should be good)
  • You might need to change the file paths, and please be sure you change the corresponding paths in the codes as well

Getting started

  1. Install dependencies
    # Imgaug (or see https://imgaug.readthedocs.io/en/latest/source/installation.html)
    conda config --add channels conda-forge
    conda install imgaug
  2. Creat synthetic sets
    # By default it creates 500 synthetic sets
    python generate_synthetic_sets.py
  3. Learn classifier on CIFAR-10 (DenseNet-10-12)
    # Save as "PROJECT_DIR/DenseNet-40-12-ss/checkpoint.pth.tar"
    # Modified based on the wonderful github of https://github.com/andreasveit/densenet-pytorch
    python train.py --layers 40 --growth 12 --no-bottleneck --reduce 1.0
  4. Test classifier on synthetic sets
    # 1) Get "PROJECT_DIR/accuracy_cls_dense_aug.npy" file
    # 2) Get "PROJECT_DIR/accuracy_ss_dense_aug.npy" file
    # 3) You will see Rank correlation and Pearsons correlation
    # 4) The absolute error of linear regression is also shown
    python test_many.py --layers 40 --growth 12 --no-bottleneck --reduce 1.0
  5. Correlation study
    # You will see correlation.pdf;
    python analyze_correlation.py
        

Citation

If you use the code in your research, please cite:

    @inproceedings{Deng:ICML2021,
      author    = {Weijian Deng and
                   Stephen Gould and
                   Liang Zheng},
      title     = {What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?},
      booktitle = {ICML},
      year      = {2021}
    }

License

MIT

Owner
Third-year PhD student at ANU.
Fully Adaptive Bayesian Algorithm for Data Analysis (FABADA) is a new approach of noise reduction methods. In this repository is shown the package developed for this new method based on \citepaper.

Fully Adaptive Bayesian Algorithm for Data Analysis FABADA FABADA is a novel non-parametric noise reduction technique which arise from the point of vi

18 Oct 20, 2022
StyleSwin: Transformer-based GAN for High-resolution Image Generation

StyleSwin This repo is the official implementation of "StyleSwin: Transformer-based GAN for High-resolution Image Generation". By Bowen Zhang, Shuyang

Microsoft 349 Dec 28, 2022
a project for 3D multi-object tracking

a project for 3D multi-object tracking

155 Jan 04, 2023
We have made you a wrapper you can't refuse

We have made you a wrapper you can't refuse We have a vibrant community of developers helping each other in our Telegram group. Join us! Stay tuned fo

20.6k Jan 09, 2023
Towards Part-Based Understanding of RGB-D Scans

Towards Part-Based Understanding of RGB-D Scans (CVPR 2021) We propose the task of part-based scene understanding of real-world 3D environments: from

26 Nov 23, 2022
A computer vision pipeline to identify the "icons" in Christian paintings

Christian-Iconography A computer vision pipeline to identify the "icons" in Christian paintings. A bit about iconography. Iconography is related to id

Rishab Mudliar 3 Jul 30, 2022
Normalizing Flows with a resampled base distribution

Resampling Base Distributions of Normalizing Flows Normalizing flows are a popular class of models for approximating probability distributions. Howeve

Vincent Stimper 24 Nov 03, 2022
Accelerated SMPL operation, commonly used in generate 3D human mesh, STAR included.

SMPL2 An enchanced and accelerated SMPL operation which commonly used in 3D human mesh generation. It takes a poses, shapes, cam_trans as inputs, outp

JinTian 20 Oct 17, 2022
An e-commerce company wants to segment its customers and determine marketing strategies according to these segments.

customer_segmentation_with_rfm Business Problem : An e-commerce company wants to

Buse Yıldırım 3 Jan 06, 2022
Codes and Data Processing Files for our paper.

Code Scripts and Processing Files for EEG Sleep Staging Paper 1. Folder Tree ./src_preprocess (data preprocessing files for SHHS and Sleep EDF) sleepE

Chaoqi Yang 18 Dec 12, 2022
The Curious Layperson: Fine-Grained Image Recognition without Expert Labels (BMVC 2021)

The Curious Layperson: Fine-Grained Image Recognition without Expert Labels Subhabrata Choudhury, Iro Laina, Christian Rupprecht, Andrea Vedaldi Code

Subhabrata Choudhury 18 Dec 27, 2022
Pytorch-Swin-Unet-V2 - a modified version of Swin Unet based on Swin Transfomer V2

Swin Unet V2 Swin Unet V2 is a modified version of Swin Unet arxiv based on Swin

Chenxu Peng 26 Dec 03, 2022
A Python package for performing pore network modeling of porous media

Overview of OpenPNM OpenPNM is a comprehensive framework for performing pore network simulations of porous materials. More Information For more detail

PMEAL 336 Dec 30, 2022
Example how to deploy deep learning model with aiohttp.

aiohttp-demos Demos for aiohttp project. Contents Imagetagger Deep Learning Image Classifier URL shortener Toxic Comments Classifier Moderator Slack B

aio-libs 661 Jan 04, 2023
The official PyTorch implementation for NCSNv2 (NeurIPS 2020)

Improved Techniques for Training Score-Based Generative Models This repo contains the official implementation for the paper Improved Techniques for Tr

174 Dec 26, 2022
Set of models for classifcation of 3D volumes

Classification models 3D Zoo - Keras and TF.Keras This repository contains 3D variants of popular CNN models for classification like ResNets, DenseNet

69 Dec 28, 2022
Hcaptcha-challenger - Gracefully face hCaptcha challenge with Yolov5(ONNX) embedded solution

hCaptcha Challenger 🚀 Gracefully face hCaptcha challenge with Yolov5(ONNX) embe

593 Jan 03, 2023
[Preprint] ConvMLP: Hierarchical Convolutional MLPs for Vision, 2021

Convolutional MLP ConvMLP: Hierarchical Convolutional MLPs for Vision Preprint link: ConvMLP: Hierarchical Convolutional MLPs for Vision By Jiachen Li

SHI Lab 143 Jan 03, 2023
SelfRemaster: SSL Speech Restoration

SelfRemaster: Self-Supervised Speech Restoration Official implementation of SelfRemaster: Self-Supervised Speech Restoration with Analysis-by-Synthesi

Takaaki Saeki 46 Jan 07, 2023
When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset of 53,000+ Legal Holdings

When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset of 53,000+ Legal Holdings This is the repository for t

RegLab 39 Jan 07, 2023