The official PyTorch code for NeurIPS 2021 ML4AD Paper, "Does Thermal data make the detection systems more reliable?"

Last update: Nov 02, 2022

Related tags

Overview

MultiModal-Collaborative (MMC) Learning Framework for integrating RGB and Thermal spectral modalities

This is the official code for NeurIPS 2021 Machine Learning for Autonomous Driving Workshop Paper, "Does Thermal data make the detection systems more reliable?" by Shruthi Gowda, Elahe Arani and Bahram Zonooz.

Methodology

Architecture

Detection Head : SSD
Detection Backbone : Resnet (CNN-based) or DEiT (Transformer-based)

MMC framework

MMC framework has multiple versions

KD.ENABLE: True
KD.ENABLE_DML: True

1. MMC (Base Version) : Det Loss + DML Loss 
    KD.DISTILL_TYPE : KL, AT, L2, L2B
    KL (KL divergence), AT (Attention loss), L2 (L2 norm at head layer), L2B (L2 norm of backbone features)
   
2. MMC v1 (Reconstruction) : Det Loss + DML Loss + Recon Loss
    KD.AUX_RECON = True
    KD.AUX_RECON_MODE = "normal"

3. MMC v2 (Cross Reconstruction) : Det Loss + DML Loss + Cross Recon Loss
    KD.AUX_RECON = True
    KD.AUX_RECON_MODE = "cross"

We also try other techniques for comparison

Fusion
1. Input Fusion
    KD.CONCAT_INPUT
2. Feature Fusion
    KD.CONCAT_FEATURES
    CONCAT_LAYERS

Installation

You can prepare the environment using:

pip install -r requirements.txt

You can build the project using the following script:

./build {conda_env_name}

Datasets

Two datasets "FLIR" and "KAIST" are used in this repo

FLIR : https://www.flir.eu/oem/adas/adas-dataset-form/

KAIST : https://soonminhwang.github.io/rgbt-ped-detection/

Running

Train

There are 2 networks, one receiving RGB images and one receiving thermal images. Both require different config files.

python train.py --config-file <thermal-config-file> --teacher-config-file <rgb-config-file>

Test

For evaluation only one network is used - the first network (RGB or Teacher network)

python test.py --config-file <config-file> --ckpt <model_final.pth>

Model Checkpoints

Cite Our Work

License

This project is licensed under the terms of the MIT license.

The official PyTorch code for NeurIPS 2021 ML4AD Paper, "Does Thermal data make the detection systems more reliable?"

Related tags

Overview

MultiModal-Collaborative (MMC) Learning Framework for integrating RGB and Thermal spectral modalities

Methodology

Architecture

MMC framework

Installation

Datasets

Running

Train

Test

Model Checkpoints

Cite Our Work

License

Owner

NeurAI

【ACMMM 2021】DSANet: Dynamic Segment Aggregation Network for Video-Level Representation Learning

A stock generator that assess a list of stocks and returns the best stocks for investing and money allocations based on users choices of volatility, duration and number of stocks

multimodal transformer

Official PyTorch implementation of "Synthesis of Screentone Patterns of Manga Characters"

PyTorch implementation for the visual prior component (i.e. perception module) of the Visually Grounded Physics Learner [Li et al., 2020].

Python calculations for the position of the sun and moon.

TensorFlow-based implementation of "Pyramid Scene Parsing Network".

PyTorch implementation of the paper:A Convolutional Approach to Melody Line Identification in Symbolic Scores.

The official PyTorch implementation of paper BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition

This is the official repository for our paper: ''Pruning Self-attentions into Convolutional Layers in Single Path''.

This is the official source code of "BiCAT: Bi-Chronological Augmentation of Transformer for Sequential Recommendation".

A colab notebook for training Stylegan2-ada on colab, transfer learning onto your own dataset.

Wider or Deeper: Revisiting the ResNet Model for Visual Recognition

LIMEcraft: Handcrafted superpixel selectionand inspection for Visual eXplanations

High dimensional black-box optimizer using Latent Action Monte Carlo Tree Search algorithm

Multi-view 3D reconstruction using neural rendering. Unofficial implementation of UNISURF, VolSDF, NeuS and more.

This repository collects 100 papers related to negative sampling methods.

Code for "Graph-Evolving Meta-Learning for Low-Resource Medical Dialogue Generation". [AAAI 2021]

Code of the lileonardo team for the 2021 Emotion and Theme Recognition in Music task of MediaEval 2021

ESTDepth: Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks (CVPR 2021)