Localizing-Visual-Sounds-the-Hard-Way

Code and Dataset for "Localizing Visual Sounds the Hard Way".

The repo contains code and our pre-trained model.

Environment

Python 3.6.8
Pytorch 1.3.0

Flickr-SoundNet

We provide the pretrained model here.

To test the model, testing data and ground truth should be downloaded from learning to localize sound source.

Then run

python test.py --data_path "path to downloaded data with structure below/" --summaries_dir "path to pretrained models" --gt_path "path to ground truth" --testset "flickr"

VGG-Sound Source

We provide the pretrained model here.

To test the model, run

python test.py --data_path "path to downloaded data with structure below/" --summaries_dir "path to pretrained models" --testset "vggss"

(Note, some gt bounding boxes are updated recently, all results on VGG-SS cause a 2~3% difference on IoU.)

Both test data should be placed in the following structure.

data path
│
└───frames
│   │   image001.jpg
│   │   image002.jpg
│   │
└───audio
    │   audio011.wav
    │   audio012.wav

Citation

@InProceedings{Chen21,
              title        = "Localizing Visual Sounds the Hard Way",
              author       = "Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, Andrew Zisserman",
              booktitle    = "CVPR",
              year         = "2021"}

Localizing Visual Sounds the Hard Way

Related tags

Overview

Localizing-Visual-Sounds-the-Hard-Way

Environment

Flickr-SoundNet

VGG-Sound Source

Citation

Owner

Honglie Chen

Recurrent Neural Network Tutorial, Part 2 - Implementing a RNN in Python and Theano

This is the repository for paper NEEDLE: Towards Non-invertible Backdoor Attack to Deep Learning Models.

Vertex AI: Serverless framework for MLOPs (ESP / ENG)

CCPD: a diverse and well-annotated dataset for license plate detection and recognition

Tensorflow AffordanceNet and AffContext implementations

GLaRA: Graph-based Labeling Rule Augmentation for Weakly Supervised Named Entity Recognition

Chinese license plate recognition

patchmatch和patchmatchstereo算法的python实现

A Python Library for Graph Outlier Detection (Anomaly Detection)

Learning embeddings for classification, retrieval and ranking.

SuRE Evaluation: A Supplementary Material

Pre-trained BERT Models for Ancient and Medieval Greek, and associated code for LaTeCH 2021 paper titled - "A Pilot Study for BERT Language Modelling and Morphological Analysis for Ancient and Medieval Greek"

Official Implementation (PyTorch) of "Point Cloud Augmentation with Weighted Local Transformations", ICCV 2021

Useful materials and tutorials for 110-1 NTU DBME5028 (Application of Deep Learning in Medical Imaging)

Data Augmentation with Variational Autoencoders

Generic U-Net Tensorflow implementation for image segmentation

A study project using the AA-RMVSNet to reconstruct buildings from multiple images

Deep Reinforcement Learning for Keras.

Pytorch tutorials for Neural Style transfert

Implementation of the SUMO (Slim U-Net trained on MODA) model