Localizing-Visual-Sounds-the-Hard-Way

Code and Dataset for "Localizing Visual Sounds the Hard Way".

The repo contains code and our pre-trained model.

Environment

Python 3.6.8
Pytorch 1.3.0

Flickr-SoundNet

We provide the pretrained model here.

To test the model, testing data and ground truth should be downloaded from learning to localize sound source.

Then run

python test.py --data_path "path to downloaded data with structure below/" --summaries_dir "path to pretrained models" --gt_path "path to ground truth" --testset "flickr"

VGG-Sound Source

We provide the pretrained model here.

To test the model, run

python test.py --data_path "path to downloaded data with structure below/" --summaries_dir "path to pretrained models" --testset "vggss"

(Note, some gt bounding boxes are updated recently, all results on VGG-SS cause a 2~3% difference on IoU.)

Both test data should be placed in the following structure.

data path
│
└───frames
│   │   image001.jpg
│   │   image002.jpg
│   │
└───audio
    │   audio011.wav
    │   audio012.wav

Citation

@InProceedings{Chen21,
              title        = "Localizing Visual Sounds the Hard Way",
              author       = "Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, Andrew Zisserman",
              booktitle    = "CVPR",
              year         = "2021"}

Localizing Visual Sounds the Hard Way

Related tags

Overview

Localizing-Visual-Sounds-the-Hard-Way

Environment

Flickr-SoundNet

VGG-Sound Source

Citation

Owner

Honglie Chen

Learning Dense Representations of Phrases at Scale (Lee et al., 2020)

Official implementation of "SinIR: Efficient General Image Manipulation with Single Image Reconstruction" (ICML 2021)

MLPs for Vision and Langauge Modeling (Coming Soon)

DEMix Layers for Modular Language Modeling

Projects of Andfun Yangon

Implementation of Diverse Semantic Image Synthesis via Probability Distribution Modeling

Official PyTorch implementation of "Adversarial Reciprocal Points Learning for Open Set Recognition"

Wordplay, an artificial Intelligence based crossword puzzle solver.

Physics-Informed Neural Networks (PINN) and Deep BSDE Solvers of Differential Equations for Scientific Machine Learning (SciML) accelerated simulation

Pytorch implementation of our method for high-resolution (e.g. 2048x1024) photorealistic video-to-video translation.

[CVPR 2020] GAN Compression: Efficient Architectures for Interactive Conditional GANs

PyTorch implementation of the paper: "Preference-Adaptive Meta-Learning for Cold-Start Recommendation", IJCAI, 2021.

Official Implementation of Swapping Autoencoder for Deep Image Manipulation (NeurIPS 2020)

DANA paper supplementary materials

The official implementation of the Interspeech 2021 paper WSRGlow: A Glow-based Waveform Generative Model for Audio Super-Resolution.

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.

Pytorch reimplementation of the Vision Transformer (An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale)

A large-scale face dataset for face parsing, recognition, generation and editing.

A method to perform unsupervised cross-region adaptation of crop classifiers trained with satellite image time series.

A pre-trained language model for social media text in Spanish