Finetuning for portuguese: NaN output from the text encoder

[lucasgris]

Hello, I am having some problems in my experiment and I would like to know if anyone have an ideia.

I am trying to finetune the LibriTTS model for portuguese. To achieve this, I am using the pretrained PLBERT model https://huggingface.co/papercup-ai/multilingual-pl-bert and the portuguese part of the CML-TTS dataset. Unfortunately, when I test the model, I get NaN output from the text encoder.

d = model.predictor.text_encoder(d, s, input_lengths, text_mask)

I am not sure if the model trained for sufficient steps, but since the training is expensive, I am not sure if I will continue. Here is a print of my training (second stage):

I have some hypotheses:

1. Needs more training
2. It is not possible to finetune the pretrained model for a different language changing only the PL-BERT (that is, it is necessary to train from scratch)
3. My configurations are not optimal, for example, I am using a batch size of only 4 because I have only one GPU to train the model.
4. Some error in the preprocessing?

What do you guys think? I am investigating, but due to limited resources it is hard to test everything.

[lucasgris]

Updating, I notice that the style and diffusion didn't start training, maybe this is the reason.

[justlab11]

I am currently trying to train a similar model and I am running into this issue as well. Due to the size of the dataset, I am still on stage 1 of training (it takes an hour to get through an epoch). Did training through stage 2 (and specifically, training the style and diffusion portions) help with this issue? Just from a rough evaluation of the model during stage 1, the text encoder produces massive values in the range 1e17.