Parallel Tacotron2
Pytorch Implementation of Google's Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling
Updates
-
2021.05.15: Implementation done. Sanity checks on training and inference. But still the model cannot converge.
I'm waiting for your contribution!Please inform me if you find any mistakes in my implementation or any valuable advice to train the model successfully. See the Implementation Issues section.
Training
Requirements
-
You can install the Python dependencies with
pip3 install -r requirements.txt
-
In addition to that, install fairseq (official document, github) to utilize
LConvBlock.
Datasets
The supported datasets:
- LJSpeech: a single-speaker English dataset consists of 13100 short audio clips of a female speaker reading passages from 7 non-fiction books, approximately 24 hours in total.
- (will be added more)
Preprocessing
After downloading the datasets, set the corpus_path in preprocess.yaml and run the preparation script:
python3 prepare_data.py config/LJSpeech/preprocess.yaml
Then, run the preprocessing script:
python3 preprocess.py config/LJSpeech/preprocess.yaml
Training
Train your model with
python3 train.py -p config/LJSpeech/preprocess.yaml -m config/LJSpeech/model.yaml -t config/LJSpeech/train.yaml
The model cannot converge yet. I'm debugging but it would be boosted if your awesome contribution is ready!
TensorBoard
Use
tensorboard --logdir output/log/LJSpeech
to serve TensorBoard on your localhost.
Implementation Issues
Overall, normalization or activation, which is not suggested in the original paper, is adequately arranged to prevent nan value (gradient) on forward and backward calculations.
Text Encoder
- Use the
FFTBlockof FastSpeech2 for the transformer block of the text encoder. - Use dropout
0.2for theConvBlockof the text encoder. - To restore "proprietary normalization engine",
- Apply the same text normalization as in FastSpeech2.
- Implement
grapheme_to_phonemefunction. (See ./text/init).
Residual Encoder
- Use
80 channelsmel-spectrogrom instead of128-bin. - Regular sinusoidal positional embedding is used in frame-level instead of combinations of three positional embeddings in Parallel Tacotron. As the model depends entirely on unsupervised learning for the position, this choice can be a reason for the fails on model converge.
Duration Predictor & Learned Upsampling (The most important but ambiguous part)
- Use log durations with the prior: there should be at least one frame in total per sequence.
- Use
nn.SiLU()for the swish activation. - When obtaining
WandC, concatenation operation is applied amongS,E, andVafter frame-domain (T domain) broadcasting ofV. As the detailed process is not described in the original paper, this choice can be a reason for the fails on model converge.
Decoder
- Use (Multi-head)
Self-attentionandLConvBlock. - Iterative mel-spectrogram is projected by a linear layer.
- Apply
nn.Tanh()to eachLConvBLockoutput (following activation pattern of decoder part in FastSpeech2).
Loss
- Use optimization & scheduler of FastSpeech2 (which is from Attention is all you need as described in the original paper).
- Base on pytorch-softdtw-cuda (post) for the soft-DTW.
- Implement customized soft-DTW in
model/soft_dtw_cuda.py, reflecting the recursion suggested in the original paper. - In the original soft-DTW, the final loss is not assumed and therefore only
Eis computed. But employed as a loss function, jacobian product is added to return target derivetive ofRw.r.t. inputX. - Currently, the maximum batch size is
6in 24GiB GPU (TITAN RTX) due to space complexity problem in soft-DTW Loss.- In the original paper, a custom differentiable diagonal band operation was implemented and used to solve the complexity of O(T^2), but this part has not been explored in the current implementation yet.
- Implement customized soft-DTW in
- For the stability, mel-spectrogroms are compressed by a sigmoid function before the soft-DTW. If the sigmoid is eliminated, the soft-DTW value is too large, producing nan in the backward.
- Guided attention loss is applied for fast convergence of the attention module in residual encoder.
Citation
@misc{lee2021parallel_tacotron2,
author = {Lee, Keon},
title = {Parallel-Tacotron2},
year = {2021},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/keonlee9420/Parallel-Tacotron2}}
}
619 Dec 14, 2022
27 Jul 07, 2022
1.1k Jan 02, 2023
36 Nov 28, 2022
10 Aug 30, 2022
1.2k Dec 26, 2022
203 Dec 30, 2022
109 Dec 27, 2022
22 Dec 13, 2022
11 Nov 27, 2022
98 Nov 16, 2022
28 Oct 10, 2022
138 Dec 22, 2022
8 Oct 04, 2022
824 Jan 07, 2023
11 Sep 23, 2022
44 Dec 20, 2022
23 Nov 11, 2022
7 May 11, 2022
56 Dec 16, 2022