Daft-Exprt - PyTorch Implementation
PyTorch Implementation of Daft-Exprt: Robust Prosody Transfer Across Speakers for Expressive Speech Synthesis
The validation logs up to 70K of synthesized mel and alignment are shown below (VCTK_val_p237-088).
Quickstart
DATASET refers to the names of datasets such as VCTK in the following documents.
Dependencies
You can install the Python dependencies with
pip3 install -r requirements.txt
Also, Dockerfile is provided for Docker users.
Inference
You have to download the pretrained models and put them in output/ckpt/DATASET/.
For a multi-speaker TTS, run
python3 synthesize.py --text "YOUR_DESIRED_TEXT" --speaker_id SPEAKER_ID --restore_step RESTORE_STEP --mode single --dataset DATASET --ref_audio REF_AUDIO
to synthesize speech with the style of input audio at REF_AUDIO. The dictionary of learned speakers can be found at preprocessed_data/VCTK/speakers.json, and the generated utterances will be put in output/result/.
Batch Inference
Batch inference is also supported, try
python3 synthesize.py --source preprocessed_data/DATASET/val.txt --restore_step RESTORE_STEP --mode batch --dataset DATASET
to synthesize all utterances consuming themselves as a reference audio in preprocessed_data/DATASET/val.txt.
Controllability
The pitch/volume/speaking rate of the synthesized utterances can be controlled by specifying the desired pitch/energy/duration ratios. For example, one can increase the speaking rate by 20 % and decrease the volume by 20 % by
python3 synthesize.py --text "YOUR_DESIRED_TEXT" --speaker_id SPEAKER_ID --restore_step RESTORE_STEP --mode single --dataset DATASET --ref_audio REF_AUDIO --duration_control 0.8 --energy_control 0.8
Training
Datasets
The supported datasets are
- VCTK: The CSTR VCTK Corpus includes speech data uttered by 110 English speakers (multi-speaker TTS) with various accents. Each speaker reads out about 400 sentences, which were selected from a newspaper, the rainbow passage and an elicitation paragraph used for the speech accent archive.
- Any of multi-speaker TTS dataset (e.g., LibriTTS) can be added following VCTK.
Preprocessing
-
For a multi-speaker TTS with external speaker embedder, download ResCNN Softmax+Triplet pretrained model of philipperemy's DeepSpeaker for the speaker embedding and locate it in
./deepspeaker/pretrained_models/. -
Run
python3 prepare_align.py --dataset DATASETfor some preparations.
For the forced alignment, Montreal Forced Aligner (MFA) is used to obtain the alignments between the utterances and the phoneme sequences. Pre-extracted alignments for the datasets are provided here. You have to unzip the files in
preprocessed_data/DATASET/TextGrid/. Alternately, you can run the aligner by yourself.After that, run the preprocessing script by
python3 preprocess.py --dataset DATASET
Training
Train your model with
python3 train.py --dataset DATASET
TensorBoard
Use
tensorboard --logdir output/log
to serve TensorBoard on your localhost. The loss curves, synthesized mel-spectrograms, and audios are shown.
Implementation Issues
- RangeParameterPredictor is built with BiLSTM rather than a single linear layer with softplus() activation (it is however implemented and named as 'range_param_predictor_paper' in GaussianUpsampling).
- Use
32batch size instead of48due to memory issues. - Use log duration instead of normal duration.
- Follow FastSpeech2 for the preprocess of pitch and energy.
- Two options for embedding for the multi-speaker TTS setting: training speaker embedder from scratch or using a pre-trained philipperemy's DeepSpeaker model (as STYLER did). You can toggle it by setting the config (between
'none'and'DeepSpeaker'). - DeepSpeaker on VCTK dataset shows clear identification among speakers. The following figure shows the T-SNE plot of extracted speaker embedding.
- For vocoder, HiFi-GAN and MelGAN are supported.
Citation
@misc{lee2021daft_exprt,
author = {Lee, Keon},
title = {Daft-Exprt},
year = {2021},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/keonlee9420/Daft-Exprt}}
}
References
- keonlee9420's WaveGrad2 for GaussianUpsampling and RangeParameterPredictor
- keonlee9420's STYLER for the (domain) adversarial training of SpeakerClassifier
- keonlee9420's StyleSpeech for reference auido interface
- FiLM: Visual Reasoning with a General Conditioning Layer
- TADAM: Task dependent adaptive metric for improved few-shot learning
![FileNotFoundError: [Errno 2] No such file or directory: './preprocessed_data/VCTK/spker_embed/p317-spker_embed.npy'](https://avatars.githubusercontent.com/u/18510825?v=4)
191 Dec 22, 2022
62 Dec 20, 2022
64 Dec 19, 2022
4.8k Jan 07, 2023
4 Sep 25, 2022
2 Oct 25, 2021
50 Dec 29, 2022
38 Dec 08, 2022
2 Nov 03, 2021
1 Jan 13, 2022
234 Dec 07, 2022
45 Oct 25, 2022
9 Jul 31, 2022
0 Oct 18, 2021
78 Dec 27, 2022
7 Oct 13, 2022
72 Jan 02, 2023
6 Dec 05, 2022
57 Dec 26, 2022
1 Mar 12, 2022