Benchmarking the robustness of Spatial-Temporal Models

Last update: Dec 16, 2022

Overview

Benchmarking the robustness of Spatial-Temporal Models

This repositery contains the code for the paper Benchmarking the Robustness of Spatial-Temporal Models Against Corruptions.

Python 2.7 and 3.7, Pytorch 1.7+, FFmpeg are required.

Requirements

pip3 install - requirements.txt

Mini Kinetics-C

Download original Kinetics400 from link.

The Mini Kinetics-C contains half of the classes in Kinetics400. All the classes can be found in mini-kinetics-200-classes.txt.

Mini Kinetics-C Leaderboard

Corruption robustness of spatial-temporal models trained on clean Mini Kinetics and evaluated on Mini Kinetics-C.

Approach	Reference	Backbone	Input Length	Sampling Method	Clean Accuracy	mPC	rPC
TimeSformer	Gedas et al.	Transformer	32	Uniform	82.2	71.4	86.9
3D ResNet	K. Hara et al.	ResNet-50	32	Uniform	73.0	59.2	81.1
I3D	J. Carreira et al.	InceptionV1	32	Uniform	70.5	57.7	81.8
SlowFast 8x4	C. Feichtenhofer at al.	ResNet-50	32	Uniform	69.2	54.3	78.5
3D ResNet	K. Hara et al.	ResNet-18	32	Uniform	66.2	53.3	80.5
TAM	Q.Fan et al.	ResNet-50	32	Uniform	66.9	50.8	75.9
X3D-M	C. Feichtenhofer	ResNet-50	32	Uniform	62.6	48.6	77.6

For fair comparison, it is recommended to submit the result of approach which follows the following settings: Backbone of ResNet-50, Input Length of 32, Uniform Sampling at Clip Level. Any result on our benchmark can be submitted via pull request.

Mini SSV2-C

Download original Something-Something-V2 datset from link.

The Mini SSV2-C contains half of the classes in Something-Something-V2. All the classes can be found in mini-ssv2-87-classes.txt.

Mini SSV2-C Leaderboard

Corruption robustness of spatial-temporal models trained on clean Mini SSV2 and evaluated on Mini SSV2-C.

Approach	Reference	Backbone	Input Length	Sampling Method	Clean Accuracy	mPC	rPC
TimeSformer	Gedas et al.	Transformer	16	Uniform	60.5	49.7	82.1
I3D	J. Carreira et al.	InceptionV1	32	Uniform	58.5	47.8	81.7
3D ResNet	K. Hara et al.	ResNet-50	32	Uniform	57.4	46.6	81.2
TAM	Q.Fan et al.	ResNet-50	32	Uniform	61.8	45.7	73.9
3D ResNet	K. Hara et al.	ResNet-18	32	Uniform	53.0	42.6	80.3
X3D-M	C. Feichtenhofer	ResNet-50	32	Uniform	49.9	40.7	81.6
SlowFast 8x4	C. Feichtenhofer at al.	ResNet-50	32	Uniform	48.7	38.4	78.8

Training and Evaluation

To help researchers reproduce the benchmark results provided in our leaderboard, we include a simple framework for training and evaluating the spatial-temporal models in the folder: benchmark_framework.

Running the code

Assume the structure of data directories is the following:

~/
  datadir/
    mini_kinetics/
      train/
        .../ (directories of class names)
          ...(hdf5 file containing video frames)
    mini_kinetics-c/
      .../ (directories of corruption names)
        .../ (directories of severity level)
          .../ (directories of class names)
            ...(hdf5 file containing video frames)

Train I3D on the Mini Kinetics dataset with 4 GPUs and 16 CPU threads (for data loading). The input lenght is 32, the batch size is 32 and learning rate is 0.01.

python3 train.py --threed_data --dataset mini_kinetics400 --frames_per_group 1 --groups 32 --logdir snapshots/ \
--lr 0.01 --backbone_net i3d -b 32 -j 16 --cuda 0,1,2,3

Test I3D on the Mini Kinetics-C dataset (pretrained model is loaded)

python3 test_corruption.py --threed_data --dataset mini_kinetics400 --frames_per_group 1 --groups 32 --logdir snapshots/ \
--pretrained snapshots/mini_kinetics400-rgb-i3d_v2-ts-max-f32-cosine-bs32-e50-v1/model_best.pth.tar --backbone_net i3d -b 32 -j 16 -e --cuda 0,1,2,3

Benchmarking the robustness of Spatial-Temporal Models

Related tags

Overview

Benchmarking the robustness of Spatial-Temporal Models

Requirements

Mini Kinetics-C

Mini Kinetics-C Leaderboard

Mini SSV2-C

Mini SSV2-C Leaderboard

Training and Evaluation

Running the code

Owner

Yi Chenyu Ian

RetinaFace: Deep Face Detection Library in TensorFlow for Python

NanoDet-Plus⚡Super fast and lightweight anchor-free object detection model. 🔥Only 980 KB(int8) / 1.8MB (fp16) and run 97FPS on cellphone🔥

A method that utilized Generative Adversarial Network (GAN) to interpret the black-box deep image classifier models by PyTorch.

Meta Learning for Semi-Supervised Few-Shot Classification

Torch-based tool for quantizing high-dimensional vectors using additive codebooks

This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies.

Code for the upcoming CVPR 2021 paper

Datasets, tools, and benchmarks for representation learning of code.

Code for Deep Single-image Portrait Image Relighting

PURE: End-to-End Relation Extraction

Homepage of paper: Paint Transformer: Feed Forward Neural Painting with Stroke Prediction, ICCV 2021.

Official code for the CVPR 2021 paper "How Well Do Self-Supervised Models Transfer?"

Generate Cartoon Images using Generative Adversarial Network

Official Pytorch implementation of the paper: "Locally Shifted Attention With Early Global Integration"

CVPR 2021

[NeurIPS 2021] COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining

An official implementation of "Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation" (ICCV 2021) in PyTorch.

This repository provides a basic implementation of our GCPR 2021 paper "Learning Conditional Invariance through Cycle Consistency"

The FIRST GANs-based omics-to-omics translation framework

Code for ICMI2020 and ICMI2021 papers: "Studying Person-Specific Pointing and Gaze Behavior for Multimodal Referencing of Outside Objects from a Moving Vehicle" and "ML-PersRef: A Machine Learning-based Personalized Multimodal Fusion Approach for Referencing Outside Objects From a Moving Vehicle"