Localizing-Visual-Sounds-the-Hard-Way

Code and Dataset for "Localizing Visual Sounds the Hard Way".

The repo contains code and our pre-trained model.

Environment

Python 3.6.8
Pytorch 1.3.0

Flickr-SoundNet

We provide the pretrained model here.

To test the model, testing data and ground truth should be downloaded from learning to localize sound source.

Then run

python test.py --data_path "path to downloaded data with structure below/" --summaries_dir "path to pretrained models" --gt_path "path to ground truth" --testset "flickr"

VGG-Sound Source

We provide the pretrained model here.

To test the model, run

python test.py --data_path "path to downloaded data with structure below/" --summaries_dir "path to pretrained models" --testset "vggss"

(Note, some gt bounding boxes are updated recently, all results on VGG-SS cause a 2~3% difference on IoU.)

Both test data should be placed in the following structure.

data path
│
└───frames
│   │   image001.jpg
│   │   image002.jpg
│   │
└───audio
    │   audio011.wav
    │   audio012.wav

Citation

@InProceedings{Chen21,
              title        = "Localizing Visual Sounds the Hard Way",
              author       = "Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, Andrew Zisserman",
              booktitle    = "CVPR",
              year         = "2021"}

Localizing Visual Sounds the Hard Way

Related tags

Overview

Localizing-Visual-Sounds-the-Hard-Way

Environment

Flickr-SoundNet

VGG-Sound Source

Citation

Owner

Honglie Chen

Simple image captioning model - CLIP prefix captioning.

This repository contains code for the paper "Disentangling Label Distribution for Long-tailed Visual Recognition", published at CVPR' 2021

Traditional deepdream with VQGAN+CLIP and optical flow. Ready to use in Google Colab

Code for the paper One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation, CVPR 2021.

General-purpose program synthesiser

a practicable framework used in Deep Learning. So far UDL only provide DCFNet implementation for the ICCV paper (Dynamic Cross Feature Fusion for Remote Sensing Pansharpening)

Empower Sequence Labeling with Task-Aware Language Model

Price-Prediction-For-a-Dream-Home - A machine learning based linear regression trained model for house price prediction.

RETRO-pytorch - Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch

A repository for the updated version of CoinRun used to collect MUGEN, a multimodal video-audio-text dataset.

A simple consistency training framework for semi-supervised image semantic segmentation

AQP is a modular pipeline built to enable the comparison and testing of different quality metric configurations.

Implementation of a protein autoregressive language model, but with autoregressive infilling objective (editing subsequences capability)

This package implements THOR: Transformer with Stochastic Experts.

The official implementation of the IEEE S&P`22 paper "SoK: How Robust is Deep Neural Network Image Classification Watermarking".

A High-Performance Distributed Library for Large-Scale Bundle Adjustment

Focal and Global Knowledge Distillation for Detectors

Rainbow DQN implementation that outperforms the paper's results on 40% of games using 20x less data 🌈

Data and analysis code for an MS on SK VOC genomes phenotyping/neutralisation assays

Scrutinizing XAI with linear ground-truth data