DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism
This repository is the official PyTorch implementation of our AAAI-2022 paper, in which we propose DiffSinger (for Singing-Voice-Synthesis) and DiffSpeech (for Text-to-Speech).
Besides, more detailed & improved code framework, which contains the implementations of FastSpeech 2, DiffSpeech and our NeurIPS-2021 work PortaSpeech is coming soon
| DiffSinger/DiffSpeech at training | DiffSinger/DiffSpeech at inference |
|---|---|
 |
 |
- Dec.01, 2021: DiffSinger was accepted by AAAI-2022.
- Sep.29, 2021: Our recent work
PortaSpeech: Portable and High-Quality Generative Text-to-Speechwas accepted by NeurIPS-2021.
- May.06, 2021: We submitted DiffSinger to Arxiv
Environments
conda create -n your_env_name python=3.8
source activate your_env_name
pip install -r requirements_2080.txt (GPU 2080Ti, CUDA 10.2)
or pip install -r requirements_3090.txt (GPU 3090, CUDA 11.4)
DiffSpeech (TTS version)
1. Data Preparation
a) Download and extract the LJ Speech dataset, then create a link to the dataset folder: ln -s /xxx/LJSpeech-1.1/ data/raw/
b) Download and Unzip the ground-truth duration extracted by MFA: tar -xvf mfa_outputs.tar; mv mfa_outputs data/processed/ljspeech/
c) Run the following scripts to pack the dataset for training/inference.
CUDA_VISIBLE_DEVICES=0 python data_gen/tts/bin/binarize.py --config configs/tts/lj/fs2.yaml
# `data/binary/ljspeech` will be generated.
2. Training Example
CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/lj_ds_beta6.yaml --exp_name xxx --reset
3. Inference Example
CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/lj_ds_beta6.yaml --exp_name xxx --reset --infer
We also provide:
- the pre-trained model of DiffSpeech;
- the pre-trained model of HifiGAN vocoder;
- the individual pre-trained model of FastSpeech 2 for the shallow diffusion mechanism in DiffSpeech;
Remember to put the pre-trained models in checkpoints directory.
About the determination of 'k' in shallow diffusion: We recommend the trick introduced in Appendix B. We have already provided the proper 'k' for Ljspeech dataset in the config files.
DiffSinger (SVS version)
0. Data Acquirement
- WIP. We will provide a form to apply for PopCS dataset.
1. Data Preparation
- WIP. Similar to DiffSpeech.
2. Training Example
CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/popcs_ds_beta6.yaml --exp_name xxx --reset
# or
CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/popcs_ds_beta6_offline.yaml --exp_name xxx --reset
3. Inference Example
CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config xxx --exp_name xxx --reset --infer
The pre-trained model for SVS will be provided recently.
Tensorboard
tensorboard --logdir_spec exp_name
 |
Mel Visualization
Along vertical axis, DiffSpeech: [0-80]; FastSpeech2: [80-160].
| DiffSpeech vs. FastSpeech 2 |
|---|
 |
 |
 |
Audio Demos
Audio samples can be found in our demo page.
We also put part of the audio samples generated by DiffSpeech+HifiGAN (marked as [P]) and GTmel+HifiGAN (marked as [G]) of test set in resources/demos_1218.
(corresponding to the pre-trained model DiffSpeech)
Citation
@misc{liu2021diffsinger,
title={DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism},
author={Jinglin Liu and Chengxi Li and Yi Ren and Feiyang Chen and Zhou Zhao},
year={2021},
eprint={2105.02446},
archivePrefix={arXiv},}
Acknowledgements
Our codes are based on the following repos:
Also thanks Keon Lee for fast implementation of our work.
![size mismatch for model.encoder_embed_tokens.weight: copying a param with shape torch.Size([62, 256]) from checkpoint, the shape in current model is torch.Size([57, 256]).](https://avatars.githubusercontent.com/u/10137?v=4)
11.4k Jan 04, 2023
13 Dec 13, 2022
30 Dec 12, 2022
1 Mar 04, 2022
730 Jan 09, 2023
154 Jan 01, 2023
78 Dec 12, 2022
38 Oct 17, 2022
2 Sep 27, 2022
99 Jan 02, 2023
129 Dec 17, 2022
415 Jan 06, 2023
1 Apr 08, 2022
48 Dec 14, 2022
169 Dec 21, 2022
902 Jan 06, 2023
66 Jul 25, 2022
80 Dec 29, 2022
121 Jan 03, 2023
3.4k Dec 27, 2022