Easy to use Audio Tagging in PyTorch

Last update: Dec 22, 2022

Overview

Audio Classification, Tagging & Sound Event Detection in PyTorch

Progress:

Model Zoo

AudioSet Pretrained Models

Model	Task	mAP ^(%)	Sample Rate ^(kHz)	Window Length	Num Mels	Fmax	Weights
CNN14	Tagging	43.1	32	1024	64	14k	download
CNN14_16k	Tagging	43.8	16	512	64	8k	download

CNN14_DecisionLevelMax	SED	38.5	32	1024	64	14k	download

Note: These models will be used as a pretrained model in the fine-tuning tasks below. Check out audioset-tagging-cnn, if you want to train on AudioSet dataset.

Fine-tuned Classification Models

Model	Dataset	Accuracy ^(%)	Sample Rate ^(kHz)	Weights
CNN14	ESC50 (Fold-5)	95.75	32	download
CNN14	FSDKaggle2018 (test)	93.56	32	download
CNN14	SpeechCommandsv1 (val/test)	96.60/96.77	32	download

Fine-tuned Tagging Models

Model	Dataset	mAP(%)	AUC	d-prime	Sample Rate ^(kHz)	Config	Weights
CNN14	FSDKaggle2019	-	-	-	32	-	-

Fine-tuned SED Models

Model	Dataset	F1	Sample Rate ^(kHz)	Config	Weights
CNN14_DecisionLevelMax	DESED	-	32	-	-

Supported Datasets

Dataset	Task	Classes	Train	Val	Test	Audio Length	Audio Spec	Size
ESC-50	Classification	50	2,000	5 folds	-	5s	44.1kHz, mono	600MB
UrbanSound8k	Classification	10	8,732	10 folds	-	<=4s	Vary	5.6GB
FSDKaggle2018	Classification	41	9,473	-	1,600	300ms~30s	44.1kHz, mono	4.6GB
SpeechCommandsv1	Classification	30	51,088	6,798	6,835	<=1s	16kHz, mono	1.4GB
SpeechCommandsv2	Classification	35	84,843	9,981	11,005	<=1s	16kHz, mono	2.3GB

FSDKaggle2019*	Tagging	80	4,970+19,815	-	4,481	300ms~30s	44.1kHz, mono	24GB
MTT*	Tagging	50	19,000	-	-	-	-	3GB

DESED*	SED	10	-	-	-	10	-	-

Notes: * datasets are not available yet. Classification dataset are treated as multi-class/single-label classification and tagging and sed datasets are treated as multi-label classification.

Dataset Structure (click to expand)

Download the dataset and prepare it into the following structure.

datasets
|__ ESC50
    |__ audio

|__ Urbansound8k
    |__ audio

|__ FSDKaggle2018
    |__ audio_train
    |__ audio_test
    |__ FSDKaggle2018.meta
        |__ train_post_competition.csv
        |__ test_post_competition_scoring_clips.csv

|__ SpeechCommandsv1/v2
    |__ bed
    |__ bird
    |__ ...
    |__ testing_list.txt
    |__ validation_list.txt

Augmentations (click to expand)

Currently, the following augmentations are supported. More will be added in the future. You can test the effects of augmentations with this notebook

WaveForm Augmentations:

Spectrogram Augmentations:

Time Masking
Frequency Masking
Filter Augmentation

Usage

Requirements (click to expand)

python >= 3.6
pytorch >= 1.8.1
torchaudio >= 0.8.1

Other requirements can be installed with pip install -r requirements.txt.

Configuration (click to expand)

Create a configuration file in configs. Sample configuration for ESC50 dataset can be found here.
Copy the contents of this and then edit the fields you think if it is needed.
This configuration file is needed for all of training, evaluation and prediction scripts.

Training (click to expand)

To train with a single GPU:

$ python tools/train.py --cfg configs/CONFIG_FILE_NAME.yaml

To train with multiple gpus, set DDP field in config file to true and run as follows:

$ python -m torch.distributed.launch --nproc_per_node=2 --use_env tools/train.py --cfg configs/CONFIG_FILE_NAME.yaml

Evaluation (click to expand)

Make sure to set MODEL_PATH of the configuration file to your trained model directory.

$ python tools/val.py --cfg configs/CONFIG_FILE.yaml

Audio Classification/Tagging Inference

Set MODEL_PATH of the configuration file to your model's trained weights.
Change the dataset name in DATASET >> NAME as your trained model's dataset.
Set the testing audio file path in TEST >> FILE.
Run the following command.

$ python tools/infer.py --cfg configs/CONFIG_FILE.yaml

## for example
$ python tools/infer.py --cfg configs/audioset.yaml

You will get an output similar to this:

Class                     Confidence
----------------------  ------------
Speech                     0.897762
Telephone bell ringing     0.752206
Telephone                  0.219329
Inside, small room         0.20761
Music                      0.0770325

Sound Event Detection Inference

Set MODEL_PATH of the configuration file to your model's trained weights.
Change the dataset name in DATASET >> NAME as your trained model's dataset.
Set the testing audio file path in TEST >> FILE.
Run the following command.

$ python tools/sed_infer.py --cfg configs/CONFIG_FILE.yaml

## for example
$ python tools/sed_infer.py --cfg configs/audioset_sed.yaml

You will get an output similar to this:

Class                     Start    End
----------------------  -------  -----
Speech                      2.2    7
Telephone bell ringing      0      2.5

The following plot will also be shown, if you set PLOT to true:

References (click to expand)

Citations (click to expand)

@misc{kong2020panns,
      title={PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition}, 
      author={Qiuqiang Kong and Yin Cao and Turab Iqbal and Yuxuan Wang and Wenwu Wang and Mark D. Plumbley},
      year={2020},
      eprint={1912.10211},
      archivePrefix={arXiv},
      primaryClass={cs.SD}
}

@misc{gong2021ast,
      title={AST: Audio Spectrogram Transformer}, 
      author={Yuan Gong and Yu-An Chung and James Glass},
      year={2021},
      eprint={2104.01778},
      archivePrefix={arXiv},
      primaryClass={cs.SD}
}

@misc{nam2021heavily,
      title={Heavily Augmented Sound Event Detection utilizing Weak Predictions}, 
      author={Hyeonuk Nam and Byeong-Yun Ko and Gyeong-Tae Lee and Seong-Hu Kim and Won-Ho Jung and Sang-Min Choi and Yong-Hwa Park},
      year={2021},
      eprint={2107.03649},
      archivePrefix={arXiv},
      primaryClass={eess.AS}
}

You might also like...

TorchMetrics is a collection of 25+ PyTorch metrics implementations and an easy-to-use API to create custom metrics.

Machine learning metrics for distributed, scalable PyTorch applications.

1.2k Jan 6, 2023

TorchFlare is a simple, beginner-friendly, and easy-to-use PyTorch Framework train your models effortlessly.

TorchFlare TorchFlare is a simple, beginner-friendly and an easy-to-use PyTorch Framework train your models without much effort. It provides an almost

85 Dec 26, 2022

A more easy-to-use implementation of KPConv based on PyTorch.

A more easy-to-use implementation of KPConv This repo contains a more easy-to-use implementation of KPConv based on PyTorch. Introduction KPConv is a

36 Dec 29, 2022

Use MATLAB to simulate the signal and extract features. Use PyTorch to build and train deep network to do spectrum sensing.

Deep-Learning-based-Spectrum-Sensing Use MATLAB to simulate the signal and extract features. Use PyTorch to build and train deep network to do spectru

10 Dec 14, 2022

Fast image augmentation library and easy to use wrapper around other libraries. Documentation: https://albumentations.ai/docs/ Paper about library: https://www.mdpi.com/2078-2489/11/2/125

Albumentations Albumentations is a Python library for image augmentation. Image augmentation is used in deep learning and computer vision tasks to inc

11.4k Jan 9, 2023

Fast, flexible and easy to use probabilistic modelling in Python.

Please consider citing the JMLR-MLOSS Manuscript if you've used pomegranate in your academic work! pomegranate is a package for building probabilistic

3k Dec 29, 2022

High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.

What is xLearn? xLearn is a high performance, easy-to-use, and scalable machine learning package that contains linear model (LR), factorization machin

3k Jan 3, 2023

High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.

What is xLearn? xLearn is a high performance, easy-to-use, and scalable machine learning package that contains linear model (LR), factorization machin

2.8k Feb 12, 2021

A fast and easy to use, moddable, Python based Minecraft server!

PyMine PyMine - The fastest, easiest to use, Python-based Minecraft Server! Features Note: This list is not always up to date, and doesn't contain all

144 Dec 30, 2022

Releases(v0.2.0)

v0.2.0(Aug 17, 2021)
This release includes the following:

Fine-tuned on ESC50, FSDKaggle2018, SpeechCommandsv1

Add waveform augmentations

Add spectrogram augmentations

Add augmentation testing notebook

Add tagging metrics

Source code(tar.gz)
Source code(zip)
v0.1.0(Aug 13, 2021)
Add the following datasets:

ESC50

UrbanSound8k

FSDKaggle2018

SpeechCommandsv1/v2

Release fine-tuned model on ESC50.
Source code(tar.gz)
Source code(zip)

Owner

sithu3

AI Developer

GitHub Repository

Using Language Model to Bootstrap Human Activity Recognition Ambient Sensors Based in Smart Homes

Using Language Model to Bootstrap Human Activity Recognition Ambient Sensors Based in Smart Homes This repository is the official implementation of Us

0 Oct 18, 2021

mPose3D, a mmWave-based 3D human pose estimation model.

35 Nov 08, 2022

Unsupervised Foreground Extraction via Deep Region Competition

Unsupervised Foreground Extraction via Deep Region Competition [Paper] [Code] The official code repository for NeurIPS 2021 paper "Unsupervised Foregr

28 Nov 06, 2022

Hierarchical Memory Matching Network for Video Object Segmentation (ICCV 2021)

Hierarchical Memory Matching Network for Video Object Segmentation Hongje Seong, Seoung Wug Oh, Joon-Young Lee, Seongwon Lee, Suhyeon Lee, Euntai Kim

72 Dec 14, 2022

Code Release for Learning to Adapt to Evolving Domains

EAML Code release for "Learning to Adapt to Evolving Domains" (NeurIPS 2020) Prerequisites PyTorch = 0.4.0 (with suitable CUDA and CuDNN version) tor

23 Dec 07, 2022

A toolkit for making real world machine learning and data analysis applications in C++

dlib C++ library Dlib is a modern C++ toolkit containing machine learning algorithms and tools for creating complex software in C++ to solve real worl

11.6k Jan 01, 2023

[ICLR 2021] HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark

HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark Accepted as a spotlight paper at ICLR 2021. Table of content File structure Prerequi

72 Jan 03, 2023

A TensorFlow implementation of SOFA, the Simulator for OFfline LeArning and evaluation.

SOFA This repository is the implementation of SOFA, the Simulator for OFfline leArning and evaluation. Keeping Dataset Biases out of the Simulation: A

22 Nov 23, 2022

Official implementation of "SinIR: Efficient General Image Manipulation with Single Image Reconstruction" (ICML 2021)

SinIR (Official Implementation) Requirements To install requirements: pip install -r requirements.txt We used Python 3.7.4 and f-strings which are in

47 Oct 11, 2022

[NeurIPS 2021] Garment4D: Garment Reconstruction from Point Cloud Sequences

Garment4D [PDF] | [OpenReview] | [Project Page] Overview This is the codebase for our NeurIPS 2021 paper Garment4D: Garment Reconstruction from Point

112 Dec 23, 2022

[ACL-IJCNLP 2021] Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning

CLNER The code is for our ACL-IJCNLP 2021 paper: Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning CLNER is a

71 Dec 08, 2022

Anatomy of Matplotlib -- tutorial developed for the SciPy conference

Introduction This tutorial is a complete re-imagining of how one should teach users the matplotlib library. Hopefully, this tutorial may serve as insp

1.1k Dec 29, 2022

Submodular Subset Selection for Active Domain Adaptation (ICCV 2021)

S3VAADA: Submodular Subset Selection for Virtual Adversarial Active Domain Adaptation ICCV 2021 Harsh Rangwani, Arihant Jain*, Sumukh K Aithal*, R. Ve

13 Dec 28, 2022

A python package simulating the quasi-2D pseudospin-1/2 Gross-Pitaevskii equation with NVIDIA GPU acceleration.

A python package simulating the quasi-2D pseudospin-1/2 Gross-Pitaevskii equation with NVIDIA GPU acceleration. Introduction spinor-gpe is high-level,

2 Sep 20, 2022

Learning Confidence for Out-of-Distribution Detection in Neural Networks

Learning Confidence Estimates for Neural Networks This repository contains the code for the paper Learning Confidence for Out-of-Distribution Detectio

235 Jan 05, 2023

MINERVA: An out-of-the-box GUI tool for offline deep reinforcement learning

MINERVA is an out-of-the-box GUI tool for offline deep reinforcement learning, designed for everyone including non-programmers to do reinforcement learning as a tool.

80 Nov 06, 2022

This project demonstrates the use of neural networks and computer vision to create a classifier that interprets the Brazilian Sign Language.

LIBRAS-Image-Classifier This project demonstrates the use of neural networks and computer vision to create a classifier that interprets the Brazilian

26 Oct 14, 2022

[CVPR'22] COAP: Learning Compositional Occupancy of People

COAP: Compositional Articulated Occupancy of People Paper | Video | Project Page This is the official implementation of the CVPR 2022 paper COAP: Lear

111 Dec 11, 2022

Danfeng Hong, Lianru Gao, Jing Yao, Bing Zhang, Antonio Plaza, Jocelyn Chanussot. Graph Convolutional Networks for Hyperspectral Image Classification, IEEE TGRS, 2021.

Graph Convolutional Networks for Hyperspectral Image Classification Danfeng Hong, Lianru Gao, Jing Yao, Bing Zhang, Antonio Plaza, Jocelyn Chanussot T

154 Dec 13, 2022

Recursive Bayesian Networks

Recursive Bayesian Networks This repository contains the code to reproduce the results from the NeurIPS 2021 paper Lieck R, Rohrmeier M (2021) Recursi

11 Oct 18, 2022

Easy to use Audio Tagging in PyTorch

Related tags

Overview

Audio Classification, Tagging & Sound Event Detection in PyTorch

Model Zoo

Supported Datasets

Usage

You might also like...

TorchMetrics is a collection of 25+ PyTorch metrics implementations and an easy-to-use API to create custom metrics.

TorchFlare is a simple, beginner-friendly, and easy-to-use PyTorch Framework train your models effortlessly.

A more easy-to-use implementation of KPConv based on PyTorch.

Use MATLAB to simulate the signal and extract features. Use PyTorch to build and train deep network to do spectrum sensing.

Fast image augmentation library and easy to use wrapper around other libraries. Documentation: https://albumentations.ai/docs/ Paper about library: https://www.mdpi.com/2078-2489/11/2/125

Fast, flexible and easy to use probabilistic modelling in Python.

High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.

High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.

A fast and easy to use, moddable, Python based Minecraft server!

Releases(v0.2.0)

v0.2.0(Aug 17, 2021)

v0.1.0(Aug 13, 2021)

Owner

sithu3

Using Language Model to Bootstrap Human Activity Recognition Ambient Sensors Based in Smart Homes

mPose3D, a mmWave-based 3D human pose estimation model.

Unsupervised Foreground Extraction via Deep Region Competition

Hierarchical Memory Matching Network for Video Object Segmentation (ICCV 2021)

Code Release for Learning to Adapt to Evolving Domains

A toolkit for making real world machine learning and data analysis applications in C++

[ICLR 2021] HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark

A TensorFlow implementation of SOFA, the Simulator for OFfline LeArning and evaluation.

Official implementation of "SinIR: Efficient General Image Manipulation with Single Image Reconstruction" (ICML 2021)

[NeurIPS 2021] Garment4D: Garment Reconstruction from Point Cloud Sequences

[ACL-IJCNLP 2021] Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning

Anatomy of Matplotlib -- tutorial developed for the SciPy conference

Submodular Subset Selection for Active Domain Adaptation (ICCV 2021)

A python package simulating the quasi-2D pseudospin-1/2 Gross-Pitaevskii equation with NVIDIA GPU acceleration.

Learning Confidence for Out-of-Distribution Detection in Neural Networks

MINERVA: An out-of-the-box GUI tool for offline deep reinforcement learning

This project demonstrates the use of neural networks and computer vision to create a classifier that interprets the Brazilian Sign Language.

[CVPR'22] COAP: Learning Compositional Occupancy of People

Danfeng Hong, Lianru Gao, Jing Yao, Bing Zhang, Antonio Plaza, Jocelyn Chanussot. Graph Convolutional Networks for Hyperspectral Image Classification, IEEE TGRS, 2021.

Recursive Bayesian Networks