SAAVN - Sound Adversarial Audio-Visual Navigation,ICLR2022 (In PyTorch)

Last update: Aug 30, 2022

Related tags

Deep Learning SAAVN

Overview

SAAVN

SAAVN Code release for paper "Sound Adversarial Audio-Visual Navigation,ICLR2022" (In PyTorch)

These code are under cleaning! Some of bugs maybe happen, please tell me if you have any trouble.

Thanks

These codes are based on the SoundSpaces code base.

Usage

This repo supports AudioGoal Task on Replica and Matterport3D datasets.

Below we show the commands for training and evaluating AudioGoal with Depth sensor on Replica, but it applies to Matterport dataset as well.

Training

python main.py --default av_nav --run-type train --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Validation (evaluate each checkpoint and generate a validation curve)

python main.py --default av_nav --run-type eval --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Test the best validation checkpoint based on validation curve

python main.py --default av_nav --run-type eval --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Generate demo video with audio

python main.py --default av_nav --run-type eval --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Note: [exp_config_file] is the main parameter configuration file of the experiment, while [tag_config_file] is special parameter configuration file for abalation experiments.

Citation

If you use this model in your research, please cite the following paper:

@inproceedings{YinfengICLR2022saavn,
	title = {Sound Adversarial Audio-Visual Navigation},
	author = {Yinfeng Yu, Wenbing Huang, Fuchun Sun, Changan Chen, Yikai Wang, Xiaohong Liu},
	year = {2022},
        booktitle={ICLR},
}

SAAVN - Sound Adversarial Audio-Visual Navigation,ICLR2022 (In PyTorch)

Related tags

Overview

SAAVN

SAAVN Code release for paper "Sound Adversarial Audio-Visual Navigation,ICLR2022" (In PyTorch)

These code are under cleaning! Some of bugs maybe happen, please tell me if you have any trouble.

Thanks

Usage

Citation

Owner

YinfengYu

i-SpaSP: Structured Neural Pruning via Sparse Signal Recovery

This is code to fit per-pixel environment map with spherical Gaussian lobes, using LBFGS optimization

The first public PyTorch implementation of Attentive Recurrent Comparators

Anchor-free Oriented Proposal Generator for Object Detection

SMCA replication There are no extra compiled components in SMCA DETR and package dependencies are minimal

AI Face Mesh: This is a simple face mesh detection program based on Artificial intelligence.

Fast and customizable reconnaissance workflow tool based on simple YAML based DSL.

COD-Rank-Localize-and-Segment (CVPR2021)

PURE: End-to-End Relation Extraction

Simple and ready-to-use tutorials for TensorFlow

a practicable framework used in Deep Learning. So far UDL only provide DCFNet implementation for the ICCV paper (Dynamic Cross Feature Fusion for Remote Sensing Pansharpening)

Interactive web apps created using geemap and streamlit

Code for DeepXML: A Deep Extreme Multi-Label Learning Framework Applied to Short Text Documents

A Python library for differentiable optimal control on accelerators.

Official implementation of SynthTIGER (Synthetic Text Image GEneratoR) ICDAR 2021

Novel and high-performance medical image classification pipelines are heavily utilizing ensemble learning strategies

Vit-ImageClassification - Pytorch ViT for Image classification on the CIFAR10 dataset

PixelPick This is an official implementation of the paper "All you need are a few pixels: semantic segmentation with PixelPick."

Code for Graph-to-Tree Learning for Solving Math Word Problems (ACL 2020)

Probabilistic Cross-Modal Embedding (PCME) CVPR 2021