efficient neural audio synthesis in the waveform domain

Last update: Dec 23, 2022

Overview

neural waveshaping synthesis

real-time neural audio synthesis in the waveform domain

paper • website • colab • audio

by Ben Hayes, Charalampos Saitis, György Fazekas

This repository is the official implementation of Neural Waveshaping Synthesis.

Model Architecture

Requirements

To install:

pip install -r requirements.txt
pip install -e .

We recommend installing in a virtual environment.

Data

We trained our checkpoints on the URMP dataset. Once downloaded, the dataset can be preprocessed using scripts/create_urmp_dataset.py. This will consolidate recordings of each instrument within the dataset and preprocess them according to the pipeline in the paper.

python scripts/create_urmp_dataset.py \
  --gin-file gin/data/urmp_4second_crepe.gin \ 
  --data-directory /path/to/urmp \
  --output-directory /path/to/output \
  --device cuda:0  # torch device string for CREPE model

Alternatively, you can supply your own dataset and use the general create_dataset.py script:

python scripts/create_dataset.py \
  --gin-file gin/data/urmp_4second_crepe.gin \ 
  --data-directory /path/to/dataset \
  --output-directory /path/to/output \
  --device cuda:0  # torch device string for CREPE model

Training

To train a model on the URMP dataset, use this command:

python scripts/train.py \
  --gin-file gin/train/train_newt.gin \
  --dataset-path /path/to/processed/urmp \
  --urmp \
  --instrument vn \  # select URMP instrument with abbreviated string
  --load-data-to-memory

Or to use a non-URMP dataset:

python scripts/train.py \
  --gin-file gin/train/train_newt.gin \
  --dataset-path /path/to/processed/data \
  --load-data-to-memory

efficient neural audio synthesis in the waveform domain

Related tags

Overview

neural waveshaping synthesis

real-time neural audio synthesis in the waveform domain

paper • website • colab • audio

Model Architecture

Requirements

Data

Training

Owner

Ben Hayes

Constrained Language Models Yield Few-Shot Semantic Parsers

[CVPR 2021] NormalFusion: Real-Time Acquisition of Surface Normals for High-Resolution RGB-D Scanning

Code accompanying the paper "Knowledge Base Completion Meets Transfer Learning"

Code, Models and Datasets for OpenViDial Dataset

Implicit Graph Neural Networks

GDSC-ML Team Interview Task

Code repository accompanying the paper "On Adversarial Robustness: A Neural Architecture Search perspective"

A super lightweight Lagrangian model for calculating millions of trajectories using ERA5 data

Real-world Anomaly Detection in Surveillance Videos- pytorch Re-implementation

HyDiff: Hybrid Differential Software Analysis

Abstractive opinion summarization system (SelSum) and the largest dataset of Amazon product summaries (AmaSum). EMNLP 2021 conference paper.

[CVPR 21] Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.

Codes for the compilation and visualization examples to the HIF vegetation dataset

Configure SRX interfaces with Scrapli

2021 National Underwater Robotics Vision Optics

VoxHRNet - Whole Brain Segmentation with Full Volume Neural Network

Implementation of "Deep Implicit Templates for 3D Shape Representation"

Real-Time High-Resolution Background Matting

Official Implementation of "LUNAR: Unifying Local Outlier Detection Methods via Graph Neural Networks"

Gradient-free global optimization algorithm for multidimensional functions based on the low rank tensor train format