efficient neural audio synthesis in the waveform domain

Last update: Dec 23, 2022

Overview

neural waveshaping synthesis

real-time neural audio synthesis in the waveform domain

paper • website • colab • audio

by Ben Hayes, Charalampos Saitis, György Fazekas

This repository is the official implementation of Neural Waveshaping Synthesis.

Model Architecture

Requirements

To install:

pip install -r requirements.txt
pip install -e .

We recommend installing in a virtual environment.

Data

We trained our checkpoints on the URMP dataset. Once downloaded, the dataset can be preprocessed using scripts/create_urmp_dataset.py. This will consolidate recordings of each instrument within the dataset and preprocess them according to the pipeline in the paper.

python scripts/create_urmp_dataset.py \
  --gin-file gin/data/urmp_4second_crepe.gin \ 
  --data-directory /path/to/urmp \
  --output-directory /path/to/output \
  --device cuda:0  # torch device string for CREPE model

Alternatively, you can supply your own dataset and use the general create_dataset.py script:

python scripts/create_dataset.py \
  --gin-file gin/data/urmp_4second_crepe.gin \ 
  --data-directory /path/to/dataset \
  --output-directory /path/to/output \
  --device cuda:0  # torch device string for CREPE model

Training

To train a model on the URMP dataset, use this command:

python scripts/train.py \
  --gin-file gin/train/train_newt.gin \
  --dataset-path /path/to/processed/urmp \
  --urmp \
  --instrument vn \  # select URMP instrument with abbreviated string
  --load-data-to-memory

Or to use a non-URMP dataset:

python scripts/train.py \
  --gin-file gin/train/train_newt.gin \
  --dataset-path /path/to/processed/data \
  --load-data-to-memory

efficient neural audio synthesis in the waveform domain

Related tags

Overview

neural waveshaping synthesis

real-time neural audio synthesis in the waveform domain

paper • website • colab • audio

Model Architecture

Requirements

Data

Training

Owner

Ben Hayes

Reference models and tools for Cloud TPUs.

Tensorflow AffordanceNet and AffContext implementations

Some code of the implements of Geological Modeling Using 3D Pixel-Adaptive and Deformable Convolutional Neural Network

A PyTorch-based open-source framework that provides methods for improving the weakly annotated data and allows researchers to efficiently develop and compare their own methods.

Manifold Alignment for Semantically Aligned Style Transfer

[ECCV2020] Content-Consistent Matching for Domain Adaptive Semantic Segmentation

Tensorflow implementation for Self-supervised Graph Learning for Recommendation

Source Code For Template-Based Named Entity Recognition Using BART

Digital Twin Mobility Profiling: A Spatio-Temporal Graph Learning Approach

ServiceX Transformer that converts flat ROOT ntuples into columnwise data

Informal Persian Universal Dependency Treebank

π-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

Source code for "Progressive Transformers for End-to-End Sign Language Production" (ECCV 2020)

Neon: an add-on for Lightbulb making it easier to handle component interactions

Supplementary code for the AISTATS 2021 paper "Matern Gaussian Processes on Graphs".

A Pytorch reproduction of Range Loss, which is proposed in paper 《Range Loss for Deep Face Recognition with Long-Tailed Training Data》

Neural Surface Maps

Continual learning with sketched Jacobian approximations

Adversarial Framework for (non-) Parametric Image Stylisation Mosaics

Semi-supervised semantic segmentation needs strong, varied perturbations