(NeurIPS 2021) Realistic Evaluation of Transductive Few-Shot Learning

Overview

Realistic evaluation of transductive few-shot learning

Introduction

This repo contains the code for our NeurIPS 2021 submitted paper "Realistic evaluation of transductive few-shot learning". This is a framework that regroups all methods evaluated in our paper except for SIB and LR-ICI. Results provided in the paper can be reproduced with this repo. Code was developed under python 3.8.3 and pytorch 1.4.0.

1. Getting started

1.1 Quick installation (recommended) (Download datasets and models)

To download datasets and pre-trained models (checkpoints), follow instructions 1.1.1 to 1.1.2 of NeurIPS 2020 paper "TIM: Transductive Information Maximization" public implementation (https://github.com/mboudiaf/TIM)

1.1.1 Place datasets

Make sure to place the downloaded datasets (data/ folder) at the root of the directory.

1.1.2 Place models

Make sure to place the downloaded pre-trained models (checkpoints/ folder) at the root of the directory.

1.2 Manual installation

Follow instruction 1.2 of NeurIPS 2020 paper "TIM: Transductive Information Maximization" public implementation (https://github.com/mboudiaf/TIM) if facing issues with previous steps. Make sure to place data/ and checkpoints/ folders at the root of the directory.

2. Requirements

To install requirements:

conda create --name <env> --file requirements.txt

Where <env> is the name of your environment

3. Reproducing the main results

Before anything, activate the environment:

source activate <env>

3.1 Table 1 and 2 results in paper

Evaluation in a 5-shot scenario on mini-Imagenet using RN-18 as backbone (Table 1. in paper)

Method 1-shot 5-shot 10-shot 20-shot
SimpleShot 63.0 80.1 84.0 86.1
PT-MAP 60.1 (↓16.8) 67.1 (↓18.2) 68.8 (↓18.0) 70.4 (↓17.4)
LaplacianShot 65.4 (↓4.7) 81.6 (↓0.5) 84.1 (↓0.2) 86.0 (↑0.5)
BDCSPN 67.0 (↓2.4) 80.2 (↓1.8) 82.7 (↓1.4) 84.6 (↓1.1)
TIM 67.3 (↓4.5) 79.8 (↓4.1) 82.3 (↓3.8) 84.2 (↓3.7)
α-TIM 67.4 82.5 85.9 87.9

To reproduce the results from Table 1. and 2. in the paper, from the root of the directory execute this python command.

python3 -m src.main --base_config <path_to_base_config_file> --method_config <path_to_method_config_file> 

The <path_to_base_config_file> follows this hierarchy:

config/<balanced or dirichlet>/base_config/<resnet18 or wideres>/<mini or tiered or cub>/base_config.yaml

The <path_to_method_config_file> follows this hierarchy:

config/<balanced or dirichlet>/methods_config/<alpha_tim or baseline or baseline_pp or bdcspn or entropy_min or laplacianshot or protonet or pt_map or simpleshot or tim>.yaml

For instance, if you want to reproduce the results in the balanced setting on mini-Imagenet, using ResNet-18, with alpha-TIM method go to the root of the directory and execute:

python3 -m src.main --base_config config/balanced/base_config/resnet18/mini/base_config.yaml --method_config config/balanced/methods_config/alpha_tim.yaml

If you want to reproduce the results in the randomly balanced setting on mini-Imagenet, using ResNet-18, with alpha-TIM method go to the root of the directory and execute:

python3 -m src.main --base_config config/dirichlet/base_config/resnet18/mini/base_config.yaml --method_config config/dirichlet/methods_config/alpha_tim.yaml

Reusable data sampler module

One of our main contribution is our realistic task sampling method following Dirichlet's distribution. plot

Our realistic sampler can be found in sampler.py file. The sampler has been implemented following Pytorch's norms and in a way that it can be easily reused and integrated in other projects.

The following notebook exemple_realistic_sampler.ipynb is an exemple that shows how to initialize and use our realistic category sampler.

Contact

For further questions or details, reach out to Olivier Veilleux ([email protected])

Acknowledgements

Special thanks to the authors of NeurIPS 2020 paper "TIM: Transductive Information Maximization" (TIM) (https://github.com/mboudiaf/TIM) for publicly sharing their pre-trained models and their source code from which this repo was inspired from.

Owner
Olivier Veilleux
Olivier Veilleux
Soft actor-critic is a deep reinforcement learning framework for training maximum entropy policies in continuous domains.

This repository is no longer maintained. Please use our new Softlearning package instead. Soft Actor-Critic Soft actor-critic is a deep reinforcement

Tuomas Haarnoja 752 Jan 07, 2023
Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather

LiDAR fog simulation Created by Martin Hahner at the Computer Vision Lab of ETH Zurich. This is the official code release of the paper Fog Simulation

Martin Hahner 110 Dec 30, 2022
Relative Human dataset, CVPR 2022

Relative Human (RH) contains multi-person in-the-wild RGB images with rich human annotations, including: Depth layers (DLs): relative depth relationsh

Yu Sun 112 Dec 02, 2022
Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.

Pretrained Language Model This repository provides the latest pretrained language models and its related optimization techniques developed by Huawei N

HUAWEI Noah's Ark Lab 2.6k Jan 01, 2023
BlueFog Tutorials

BlueFog Tutorials Welcome to the BlueFog tutorials! In this repository, we've put together a collection of awesome Jupyter notebooks. These notebooks

4 Oct 27, 2021
Data and codes for ACL 2021 paper: Towards Emotional Support Dialog Systems

Emotional-Support-Conversation Copyright © 2021 CoAI Group, Tsinghua University. All rights reserved. Data and codes are for academic research use onl

126 Dec 21, 2022
Making Structure-from-Motion (COLMAP) more robust to symmetries and duplicated structures

SfM disambiguation with COLMAP About Structure-from-Motion generally fails when the scene exhibits symmetries and duplicated structures. In this repos

Computer Vision and Geometry Lab 193 Dec 26, 2022
This is the official code for the paper "Tracker Meets Night: A Transformer Enhancer for UAV Tracking".

SCT This is the official code for the paper "Tracker Meets Night: A Transformer Enhancer for UAV Tracking" The spatial-channel Transformer (SCT) enhan

Intelligent Vision for Robotics in Complex Environment 27 Nov 23, 2022
4th place solution for the SIGIR 2021 challenge.

SIGIR-2021 (Tinkoff.AI) How to start Download train and test data: https://sigir-ecom.github.io/data-task.html Place it under sigir-2021/data/. Run py

Tinkoff.AI 4 Jul 01, 2022
Rank 3 : Source code for OPPO 6G Data Generation Challenge

OPPO 6G Data Generation with an E2E Framework Homepage of OPPO 6G Data Generation Challenge Datasets H1_32T4R.mat H2_32T4R.mat Please put the original

Sen Pei 97 Jan 07, 2023
The Instructed Glacier Model (IGM)

The Instructed Glacier Model (IGM) Overview The Instructed Glacier Model (IGM) simulates the ice dynamics, surface mass balance, and its coupling thro

27 Dec 16, 2022
The best solution of the Weather Prediction track in the Yandex Shifts challenge

yandex-shifts-weather The repository contains information about my solution for the Weather Prediction track in the Yandex Shifts challenge https://re

Ivan Yu. Bondarenko 15 Dec 18, 2022
A Python Package for Portfolio Optimization using the Critical Line Algorithm

PyCLA A Python Package for Portfolio Optimization using the Critical Line Algorithm Getting started To use PyCLA, clone the repo and install the requi

19 Oct 11, 2022
A fast poisson image editing implementation that can utilize multi-core CPU or GPU to handle a high-resolution image input.

Poisson Image Editing - A Parallel Implementation Jiayi Weng (jiayiwen), Zixu Chen (zixuc) Poisson Image Editing is a technique that can fuse two imag

Jiayi Weng 110 Dec 27, 2022
Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time

Semi Hand-Object Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time (CVPR 2021).

96 Dec 27, 2022
The source code for CATSETMAT: Cross Attention for Set Matching in Bipartite Hypergraphs

catsetmat The source code for CATSETMAT: Cross Attention for Set Matching in Bipartite Hypergraphs To be able to run it, add catsetmat to PYTHONPATH H

2 Dec 19, 2022
Revisiting, benchmarking, and refining Heterogeneous Graph Neural Networks.

Heterogeneous Graph Benchmark Revisiting, benchmarking, and refining Heterogeneous Graph Neural Networks. Roadmap We organize our repo by task, and on

THUDM 176 Dec 17, 2022
Official Implementation for the paper DeepFace-EMD: Re-ranking Using Patch-wise Earth Mover’s Distance Improves Out-Of-Distribution Face Identification

DeepFace-EMD: Re-ranking Using Patch-wise Earth Mover’s Distance Improves Out-Of-Distribution Face Identification Official Implementation for the pape

Anh M. Nguyen 36 Dec 28, 2022
Implementation of Gans

GAN Generative Adverserial Networks are an approach to generative data modelling using Deep learning methods. I have currently implemented : DCGAN on

Sibam Parida 5 Sep 07, 2021
In the case of your data having only 1 channel while want to use timm models

timm_custom Description In the case of your data having only 1 channel while want to use timm models (with or without pretrained weights), run the fol

2 Nov 26, 2021