efficient neural audio synthesis in the waveform domain

Last update: Dec 23, 2022

Overview

neural waveshaping synthesis

real-time neural audio synthesis in the waveform domain

paper • website • colab • audio

by Ben Hayes, Charalampos Saitis, György Fazekas

This repository is the official implementation of Neural Waveshaping Synthesis.

Model Architecture

Requirements

To install:

pip install -r requirements.txt
pip install -e .

We recommend installing in a virtual environment.

Data

We trained our checkpoints on the URMP dataset. Once downloaded, the dataset can be preprocessed using scripts/create_urmp_dataset.py. This will consolidate recordings of each instrument within the dataset and preprocess them according to the pipeline in the paper.

python scripts/create_urmp_dataset.py \
  --gin-file gin/data/urmp_4second_crepe.gin \ 
  --data-directory /path/to/urmp \
  --output-directory /path/to/output \
  --device cuda:0  # torch device string for CREPE model

Alternatively, you can supply your own dataset and use the general create_dataset.py script:

python scripts/create_dataset.py \
  --gin-file gin/data/urmp_4second_crepe.gin \ 
  --data-directory /path/to/dataset \
  --output-directory /path/to/output \
  --device cuda:0  # torch device string for CREPE model

Training

To train a model on the URMP dataset, use this command:

python scripts/train.py \
  --gin-file gin/train/train_newt.gin \
  --dataset-path /path/to/processed/urmp \
  --urmp \
  --instrument vn \  # select URMP instrument with abbreviated string
  --load-data-to-memory

Or to use a non-URMP dataset:

python scripts/train.py \
  --gin-file gin/train/train_newt.gin \
  --dataset-path /path/to/processed/data \
  --load-data-to-memory

efficient neural audio synthesis in the waveform domain

Related tags

Overview

neural waveshaping synthesis

real-time neural audio synthesis in the waveform domain

paper • website • colab • audio

Model Architecture

Requirements

Data

Training

Owner

Ben Hayes

C3d-pytorch - Pytorch porting of C3D network, with Sports1M weights

Image Super-Resolution Using Very Deep Residual Channel Attention Networks

The code for replicating the experiments from the LFI in SSMs with Unknown Dynamics paper.

Robotics environments

Official code repository for the EMNLP 2021 paper

Collection of machine learning related notebooks to share.

Ladder Variational Autoencoders (LVAE) in PyTorch

Age Progression/Regression by Conditional Adversarial Autoencoder

PyContinual (An Easy and Extendible Framework for Continual Learning)

the code for paper "Energy-Based Open-World Uncertainty Modeling for Confidence Calibration"

Official code of paper "PGT: A Progressive Method for Training Models on Long Videos" on CVPR2021

TorchGeo is a PyTorch domain library, similar to torchvision, that provides datasets, transforms, samplers, and pre-trained models specific to geospatial data.

Invert and perturb GAN images for test-time ensembling

Image-Stitching - Panorama composition using SIFT Features and a custom implementaion of RANSAC algorithm

tsflex - feature-extraction benchmarking

Retinal Vessel Segmentation with Pixel-wise Adaptive Filters (ISBI 2022)

Implementation for the "Surface Reconstruction from 3D Line Segments" paper.

MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification

Revisiting Self-Training for Few-Shot Learning of Language Model.

DeFMO: Deblurring and Shape Recovery of Fast Moving Objects (CVPR 2021)