Parallel Tacotron2
Pytorch Implementation of Google's Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling
Updates
-
2021.05.15: Implementation done. Sanity checks on training and inference. But still the model cannot converge.
I'm waiting for your contribution!Please inform me if you find any mistakes in my implementation or any valuable advice to train the model successfully. See the Implementation Issues section.
Training
Requirements
-
You can install the Python dependencies with
pip3 install -r requirements.txt
-
In addition to that, install fairseq (official document, github) to utilize
LConvBlock.
Datasets
The supported datasets:
- LJSpeech: a single-speaker English dataset consists of 13100 short audio clips of a female speaker reading passages from 7 non-fiction books, approximately 24 hours in total.
- (will be added more)
Preprocessing
After downloading the datasets, set the corpus_path in preprocess.yaml and run the preparation script:
python3 prepare_data.py config/LJSpeech/preprocess.yaml
Then, run the preprocessing script:
python3 preprocess.py config/LJSpeech/preprocess.yaml
Training
Train your model with
python3 train.py -p config/LJSpeech/preprocess.yaml -m config/LJSpeech/model.yaml -t config/LJSpeech/train.yaml
The model cannot converge yet. I'm debugging but it would be boosted if your awesome contribution is ready!
TensorBoard
Use
tensorboard --logdir output/log/LJSpeech
to serve TensorBoard on your localhost.
Implementation Issues
Overall, normalization or activation, which is not suggested in the original paper, is adequately arranged to prevent nan value (gradient) on forward and backward calculations.
Text Encoder
- Use the
FFTBlockof FastSpeech2 for the transformer block of the text encoder. - Use dropout
0.2for theConvBlockof the text encoder. - To restore "proprietary normalization engine",
- Apply the same text normalization as in FastSpeech2.
- Implement
grapheme_to_phonemefunction. (See ./text/init).
Residual Encoder
- Use
80 channelsmel-spectrogrom instead of128-bin. - Regular sinusoidal positional embedding is used in frame-level instead of combinations of three positional embeddings in Parallel Tacotron. As the model depends entirely on unsupervised learning for the position, this choice can be a reason for the fails on model converge.
Duration Predictor & Learned Upsampling (The most important but ambiguous part)
- Use log durations with the prior: there should be at least one frame in total per sequence.
- Use
nn.SiLU()for the swish activation. - When obtaining
WandC, concatenation operation is applied amongS,E, andVafter frame-domain (T domain) broadcasting ofV. As the detailed process is not described in the original paper, this choice can be a reason for the fails on model converge.
Decoder
- Use (Multi-head)
Self-attentionandLConvBlock. - Iterative mel-spectrogram is projected by a linear layer.
- Apply
nn.Tanh()to eachLConvBLockoutput (following activation pattern of decoder part in FastSpeech2).
Loss
- Use optimization & scheduler of FastSpeech2 (which is from Attention is all you need as described in the original paper).
- Base on pytorch-softdtw-cuda (post) for the soft-DTW.
- Implement customized soft-DTW in
model/soft_dtw_cuda.py, reflecting the recursion suggested in the original paper. - In the original soft-DTW, the final loss is not assumed and therefore only
Eis computed. But employed as a loss function, jacobian product is added to return target derivetive ofRw.r.t. inputX. - Currently, the maximum batch size is
6in 24GiB GPU (TITAN RTX) due to space complexity problem in soft-DTW Loss.- In the original paper, a custom differentiable diagonal band operation was implemented and used to solve the complexity of O(T^2), but this part has not been explored in the current implementation yet.
- Implement customized soft-DTW in
- For the stability, mel-spectrogroms are compressed by a sigmoid function before the soft-DTW. If the sigmoid is eliminated, the soft-DTW value is too large, producing nan in the backward.
- Guided attention loss is applied for fast convergence of the attention module in residual encoder.
Citation
@misc{lee2021parallel_tacotron2,
author = {Lee, Keon},
title = {Parallel-Tacotron2},
year = {2021},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/keonlee9420/Parallel-Tacotron2}}
}
50 Oct 08, 2021
3 Nov 23, 2022
671 Jan 03, 2023
117 Nov 27, 2022
3 Dec 02, 2022
247 Dec 27, 2022
24 Oct 23, 2022
65 Oct 04, 2022
1 Nov 30, 2021
910 Dec 30, 2022
405 Nov 27, 2022
113 Dec 14, 2022
4 Mar 07, 2022
85 Dec 22, 2022
20 Dec 21, 2022
50 Nov 17, 2022
753 Dec 17, 2022
653 Dec 19, 2022
242 Dec 30, 2022