Combining Diverse Feature Priors

Related tags

Deep Learningcopriors
Overview

Combining Diverse Feature Priors

This repository contains code for reproducing the results of our paper.

Paper: https://arxiv.org/abs/2110.08220

Blog Post: http://gradientscience.org/copriors/

Important files:

Scripts:
  pretrain_model.py: a script to pre-train the models on just the labeled data
  cotrain.py: a script to co-train pretrained model(s)
  sweep_final_models.py: a script to evaluate intermediate eras for a previously run cotrain
  
File Structure:
  datasets:
    datasets.py: the definition of the labeled/unlabeled/validation/test sets for our datasets
    transforms.py: describes the different prior transforms and spurious tinting
    co_training.py: contains the logic for model pre-training and co-training
   models:
    bagnet_custom.py: the architecture for the bagnets used in this paper
    model_utils.py: utilities for loading and building models

To generate the pre-trained priors, run:

python pretrain_model.py --dataset <DATASET NAME> --data-path <DATA PATH> --use_val --out-dir <OUTPUT PATH NAME> --arch <ARCHITECTURE NAME> --epochs 300 --lr <LR> --step_lr <STEP LR> --step_lr_gamma <STEP LR GAMMA> --additional-transform <TRANSFORM TYPE>

datasets: STLSub10, cifarsmallsub, celebaskewed 
data-path: use torchvision datasets from https://pytorch.org/vision/stable/index.html
use-val: determines whether to use validation or test set for tensorboard metrics
arch: vgg16_bn, bagnetcustom32 (bagnet for CIFAR), bagnetcustom96thin (bagnet for celeba/stl10)
lr, step-lr, step-lr-gamma are hyperparameters who's exact values can be found in our appendix.
additional-transform: which prior to use. possibilities are NONE, CANNY, SOBEL (use NONE and a bagnet architecture for the bagnet prior)

Add --spurious TINT to train with a tint (as in the tinted STL-10 experiments)

After generating the priors, the models can be self (include one prior directory) or co-trained (include both prior directories) by running:

python cotrain.py --dataset <DATASET NAME> --data-path <DATA PATH> --out-dir <OUTPUT PATH> --input-dirs <PRIOR DIRECTORY 1> --input-dirs <PRIOR DIRECTORY 2> --epochs_per_era 300 --fraction 0.05 --eras 20 --epochs 400 --arch vgg16_bn --additional-transform NONE --lr <LR> --step_lr <STEP LR> --step_lr_gamma <STEP LR GAMMA> --strategy STANDARD_CONSTANT 

This command will self/co-train the input prior directories, saving a checkpoint for each era, and then finally train a standard model on the pseudo-labels after the eras are complete.

To use the pure co-training strategy, add --pure
To use tinting as in the STL-10 tinting experiments
Owner
Madry Lab
Towards a Principled Science of Deep Learning
Madry Lab
Cosine Annealing With Warmup

CosineAnnealingWithWarmup Formulation The learning rate is annealed using a cosine schedule over the course of learning of n_total total steps with an

zhuyun 4 Apr 18, 2022
SustainBench: Benchmarks for Monitoring the Sustainable Development Goals with Machine Learning

Datasets | Website | Raw Data | OpenReview SustainBench: Benchmarks for Monitoring the Sustainable Development Goals with Machine Learning Christopher

67 Dec 17, 2022
Companion repo of the UCC 2021 paper "Predictive Auto-scaling with OpenStack Monasca"

Predictive Auto-scaling with OpenStack Monasca Giacomo Lanciano*, Filippo Galli, Tommaso Cucinotta, Davide Bacciu, Andrea Passarella 2021 IEEE/ACM 14t

Giacomo Lanciano 0 Dec 07, 2022
Supervised domain-agnostic prediction framework for probabilistic modelling

A supervised domain-agnostic framework that allows for probabilistic modelling, namely the prediction of probability distributions for individual data

The Alan Turing Institute 112 Oct 23, 2022
Neural models of common sense. 🤖

Unicorn on Rainbow Neural models of common sense. This repository is for the paper: Unicorn on Rainbow: A Universal Commonsense Reasoning Model on a N

AI2 60 Jan 05, 2023
FedScale: Benchmarking Model and System Performance of Federated Learning

FedScale: Benchmarking Model and System Performance of Federated Learning (Paper) This repository contains scripts and instructions of building FedSca

268 Jan 01, 2023
Code for "Contextual Non-Local Alignment over Full-Scale Representation for Text-Based Person Search"

Contextual Non-Local Alignment over Full-Scale Representation for Text-Based Person Search This is an implementation for our paper Contextual Non-Loca

Tencent YouTu Research 50 Dec 03, 2022
[NeurIPS 2021] A weak-shot object detection approach by transferring semantic similarity and mask prior.

TransMaS This repository is the official pytorch implementation of the following paper: NIPS2021 Mixed Supervised Object Detection by TransferringMask

BCMI 49 Jul 27, 2022
This is the official repository of the paper Stocastic bandits with groups of similar arms (NeurIPS 2021). It contains the code that was used to compute the figures and experiments of the paper.

Experiments How to reproduce experimental results of Stochastic bandits with groups of similar arms submitted paper ? Section 5 of the paper To reprod

Fabien 0 Oct 25, 2021
Official Pytorch Implementation of Length-Adaptive Transformer (ACL 2021)

Length-Adaptive Transformer This is the official Pytorch implementation of Length-Adaptive Transformer. For detailed information about the method, ple

Clova AI Research 93 Dec 28, 2022
Accelerating BERT Inference for Sequence Labeling via Early-Exit

Sequence-Labeling-Early-Exit Code for ACL 2021 paper: Accelerating BERT Inference for Sequence Labeling via Early-Exit Requirement: Please refer to re

李孝男 23 Oct 14, 2022
Complete system for facial identity system

Complete system for facial identity system. Include one-shot model, database operation, features visualization, monitoring

4 May 02, 2022
This is a Pytorch implementation of paper: DropEdge: Towards Deep Graph Convolutional Networks on Node Classification

DropEdge: Towards Deep Graph Convolutional Networks on Node Classification This is a Pytorch implementation of paper: DropEdge: Towards Deep Graph Con

401 Dec 16, 2022
MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts (ICLR 2022)

MetaShift: A Dataset of Datasets for Evaluating Distribution Shifts and Training Conflicts This repo provides the PyTorch source code of our paper: Me

88 Jan 04, 2023
DeLag: Detecting Latency Degradation Patterns in Service-based Systems

DeLag: Detecting Latency Degradation Patterns in Service-based Systems Replication package of the work "DeLag: Detecting Latency Degradation Patterns

SEALABQualityGroup @ University of L'Aquila 2 Mar 24, 2022
Pytorch implementation of FlowNet by Dosovitskiy et al.

FlowNetPytorch Pytorch implementation of FlowNet by Dosovitskiy et al. This repository is a torch implementation of FlowNet, by Alexey Dosovitskiy et

Clément Pinard 762 Jan 02, 2023
Learned model to estimate number of distinct values (NDV) of a population using a small sample.

Learned NDV estimator Learned model to estimate number of distinct values (NDV) of a population using a small sample. The model approximates the maxim

2 Nov 21, 2022
The AugNet Python module contains functions for the fast computation of image similarity.

AugNet AugNet: End-to-End Unsupervised Visual Representation Learning with Image Augmentation arxiv link In our work, we propose AugNet, a new deep le

Ming 74 Dec 28, 2022
A graph adversarial learning toolbox based on PyTorch and DGL.

GraphWar: Arms Race in Graph Adversarial Learning NOTE: GraphWar is still in the early stages and the API will likely continue to change. 🚀 Installat

Jintang Li 54 Jan 05, 2023
Semantic similarity computation with different state-of-the-art metrics

Semantic similarity computation with different state-of-the-art metrics Description • Installation • Usage • License Description TaxoSS is a semantic

6 Jun 22, 2022