Training about LRS2 dataset

[Physicsmile]

Hello, when I use your code training on the LRS2 dataset, I find that it's hard to converge. However, I use the code and config you released and don't make any modifications. I use the 'num_training_steps' written in your config which is 450,000. So far, I have trained for 300 epochs, but it still hasn't converged and the decoder_acc_val is not good(around 0.7). But I find you said you only train 200 epochs. So I wonder why this is the case.
Sorry to bother you, looking forward to your reply!

[snoop2head]

Hello @Physicsmile !
Thank you for filing the issue. Before we begin, may I know the number of GPUs and total batch size?

[Physicsmile]

Hello @Physicsmile ! Thank you for filing the issue. Before we begin, may I know the number of GPUs and total batch size?
Thank you for your reply!
I use eight A6000 GPUs, and batch size=16(same as your batch size in lrs2.yaml). Actually, except set 'resume_from_checkpoint' empty, I don't change other settings in lrs2.yaml.

[snoop2head]

@Physicsmile
Mind if I take a look at the preprocessed video examples and specifics of the loss (loss_ctc_val, loss_att_val, or loss_audio_val)?

[Physicsmile]

Sorry I don't record loss_audio_val(same as the released code), but other loss show as below:

It seems overfitting, but it appears overfitting before 200epoch. So I just wait decoder_acc_val to grow more, because it seems not good now(300 epochs).

And the preprocessed video examples information as follows:

I use /LRS/video/preprocess/prepare_LRS2.py to preprocess LRS2, not change other things.

Sorry to bother you

[snoop2head]

@Physicsmile I am actually thankful that you are showing interest in our work 🤗

• Including LRS2, we train sentence-level VSR models for 100 epochs only (as it says in our paper). I think you don't have to run more than that.
• It seems like the data preprocessing is fine.
• resume_from_checkpoint on the config lrs2.yaml file is for the evaluation purpose, so you are right about making that part empty

But may I know why the training loss is too low...? Because in case of most of our runs, the loss_step hardly drops below 30. How does the trend of loss_audio_step look like?

[Physicsmile]

I'm extremely grateful for your reply.

I can't reach the results in the paper so I just look almost issues and find you said training 200 epochs on LRS2, but forget what said in the paper, sorry for that.
I trained 70 epochs several days ago, and also remained top-10 ckpt to averge and used to test, but even so I just got 39.17% WER on LRS2. So I wonder if is the num of epoch is not enough. I remember you said in one of the issues that you can reach more than 78 on decoder_acc_val during traning, but I can't reach that high. Just as the picture above, I get 67% on decoder_acc_val if I just train models for 100 epochs and it is highly probable that perform poorly on LRS2-test based on my experience.

As for the training loss, if I just train 100 epochs, it is around 20 to 25( loss_step above is the results of training for 300 epochs, so it is lower than 100 epochs).

loss_audio_stepis shown as follows:

[snoop2head]

@Physicsmile
I see. Except for the audio reconstruction loss, the current training specifics exactly follow AutoAVSR's "3.3. Implementation details" except for otherwise stated in our paper. Maybe you can take a look into that?

[Physicsmile]

Thanks for the reminder. I'm also currently conducting the training of AutoAVSR at the same time. And I still have some I doubts about SyncVSR at present, and I hope to get them answered:

1. You train sentence-level VSR models for 100 epochs, but I am confused that with num_training_steps written in your config which is 450,000. When I use this config, 450,000 steps is more than 100 epochs, so I wonder how to set training epochs to 100 accurately.
2. I have just tried to test the above trained model on LRS2 test, but it only get 39.71% WER 😭(I even choose top-10 ckpt based on decoder_acc_val to average them and test, instead of SyncVSR orginal code choosing the best one to test). I don't know why. May I ask your model trained only on LRS2, so that I can take inference to check if there are any problems on my dataset?
3. May I ask the test WER on LRS2 of using wav2vec2.0 as quantified methods(because I have used wav2vec2.0 for training to avoid fairseq)? I can't find that on paper or issues.
4. Compared with AutoAVSR, I find that your preprocessed method is a little different from it. On the one hand, AutoAVSR first cuts the extremely long pre-training samples into segments with less than 600 frames. Then, during the training process, it will only read these already cut segments. SyncVSR randomly cut pre-train samples into segments during training. On the other hand, AutoAVSR take 96x96 size of video samples as inputs and randomly crop to 88x88, while SyncVSR take 128x128 size of samples and randomresizecrop to 96x96. And there are also some change about sampling strategy and configure_optimizers. So may I ask the considerations for your making these changes and whether these changes will have a certain impact on the results?

Since I'm really interested in your work, I'd like to discuss more with you. I would be extremely grateful if my questions could be answered. 🌹🌹🌹

[snoop2head]

@Physicsmile

Training Epochs

Apologies for the earlier confusion. To clarify, our experiments on LRS2 and LRS3 actually ran for 150 epochs, not 100 as previously stated. The reference to 100 epochs applies specifically to the joint pretraining on LRS2 + LRS3 + Vox2, which is part of a separate set of experiments we included in the paper.

Wav2Vec2.0 Results on LRS2

Our ablations involved:

• A different training framework (JAX/Flax),
• A simpler model architecture (vanilla Transformers vs. Conformer),
• A larger global batch size (128),
• Training on different hardware (TPUs),
• And notably, training without a CTC loss component.

Although we observed significant performance improvements of our method over the baseline under those conditions, these differences (especially the omission of CTC loss) meant the results were not directly comparable to our main experiments, so we did not include them in the paper. Unfortunately, we don’t have an official reported WER for a pure Wav2Vec2.0-based setup on LRS2 under the same conditions. We mainly stuck with vq-wav2vec, so I am unsure of how would it be like in your scenario.

Similarities with AutoAVSR

You are correct about using a single best checkpoint for evaluation. Unfortunately, I cannot pinpoint the exact reason for the performance discrepancy. I suggest revisiting the training scheme described in AutoAVSR—for instance, ensuring the visual frontend is set up as instructed in that paper—to see if it aligns with your implementation.

Differences from AutoAVSR Preprocessing

You are correct about the key differences in our approach:

• Segment Cutting Strategy: Rather than pre-splitting segments to fewer than 600 frames, we randomly sample segments during training. Our rationale is that the learned positional embeddings can take advantage of this random starting frame.
• Input Cropping/Resizing: We apply a random-resize-crop from 128×128 down to 96×96, whereas AutoAVSR first uses 96×96 frames and then randomly crops to 88×88. In our internal experiments, particularly on LRW, beginning with a slightly larger region of interest (128×128) and then randomly resizing led to performance gains.

I hope this helps clear up any confusion! If you have more questions or want to discuss further details—be it training or preprocessing—feel free to let me know. I’m happy to dive deeper into any part of the pipeline.

Thanks again for your interest!

[Physicsmile]

@snoop2head
Thanks for your patient reply. After carefully reading, I clear up some confusions and still have something uncertain:

1. Since the released config(lrs2.yaml) only uses num_training_steps to control training time, which is set to 450,000(more than 400 epochs actually), so I still don't know how to set training epochs to certain number(such as 150 epochs). So May I ask the lrs2.yaml using for training(since the released one is used for evaluation as you mentioned above)?
2. After carefully check about my code, I think there should be no problem with my code because I only use the released SyncVSR code. I also revisit AutoAVSR and I find visual frontend is same as the implementation in SyncVSR on LRS2, So I think I have already aligned with both AutoAVSR and SyncVSR. The only change is that I change the loggers from Wandb to TensorBoard, but I think this change won't cause any influence on the model. So I guess the only problem may be caused by my dataset. I hope get the released SyncVSR ckpt trained only on LRS2 so that I can inference on my dataset to check if my data is wrong. Could you please upload this model weights?

[Physicsmile]

@snoop2head
Hello! I wonder if it would be convenient for you recently. I would greatly appreciate your response at your earliest convenience.🌹🌹🌹

[snoop2head]

@Physicsmile

Regarding the batch size and training steps

Our implementation relies on PyTorch Lightning’s DDP (Distributed Data Parallel) backend. We used an effective batch size of 64, derived from a per-device batch size of 16 across 4 devices, with a base learning rate of 1e-3.

I recently noticed that the released version omitted an important detail: repeated sampling. LRS2 contains 140K utterances, but we performed repeated sampling for clips longer than 160 frames (the maximum length in the training split). I've now added repeated sampling—thank you for bringing this to my attention. You can see the relevant code here:

SyncVSR/LRS/video/datamodule/data_module.py

Lines 62 to 68 in 1fe0ffa

	self.train_filenames = []
	pattern = f"/data/{self.cfg.dataset}/train/*/*.pkl"
	for filename in tqdm(glob(pattern), desc="Repeated sampling for video clips longer than 160 frames..."):
	data = torch.load(filename)
	video_length = len(data["video"])
	repeat_count = int(np.ceil(video_length / 160))
	self.train_filenames.extend([filename] * repeat_count)

Regarding your mention of 400 epochs, I’m not entirely sure how that figure was obtained. Here’s our calculation: with repeated sampling, we end up with approximately 190K samples. At an effective batch size of 64, one epoch corresponds to around 3,000 steps. Therefore, training for 150 epochs would involve roughly 450K steps in total.

AutoAVSR implementation

I think your data is fine if you were to use the same preprocessing script. But please modify inference config lrs2.yaml file for proper training. Refer to the AutoAVSR's training guide on 3.3. Implementation details and resnet18_dctcn_video for frontend module.

I think I gave sufficient information for replication, but if you have more questions or want to discuss further details feel free to let me know. I’m happy to dive deeper into any part of the pipeline.

Thanks again for your interest!

[Physicsmile]

@snoop2head

Thanks for your reminder. I'm training on the newly released version and use four GPUs same as you mentioned, it will take 30 minutes every epoch so I have to wait several days to check the results. Would you mind not closing this issue for now? I hope to have further discussions with you later.
Meanwhile, I still have some doubts:

1. I have checked the maximum length in the LRS2 training split, it is 154, while the max length in ./datamodule/video_length.npy is 155 which is used to distinguished from pretrain samples. And your newly released code takes 160 as the maximum length to repeated sampling. So I'm confused about these numbers. Is 160 chosen to be the approximate number for rounding？
2. I revisit AutoAVSR's training guide on 3.3. Implementation details, it said "For our audio- and visual-only ASR models, we use a ResNet- based front-end module pre-trained on LRW". So I wonder if the frontend of SyncVSR is initialised from the model pre-trained on LRW(resnet18_dctcn_video) then trained on LRS2.
3. The structure of frontend module I used is same as SyncVSR, and it is also same as resnet18_dctcn_video(AutoAVSR), So I think there is no problem with my model.

[snoop2head]

@Physicsmile

Without our audio reconstruction loss, did you replicate AutoAVSR or CM-Aux baseline? Does the audio reconstruction loss diminish the baseline performance, or does it enhance it? These are important questions to address. AutoAVSR is an extension of Visual Speech Recognition for Multiple Languages in the Wild (=CM-Aux) but training more on synthetically generated transcripts. I think the current performance you've reported is below than that of such baseline, so I’m uncertain how best to assist in this context.

When replicating the baseline without audio reconstruction loss, I recommend referring to the Experimental settings of End-to-end Audio-visual Speech Recognition with Conformers (=CM-Seq2Seq) which is a setup for both CM-Aux and AutoAVSR. Check if you can replicate the performance of CM-Aux baseline.

Regarding the recent change, please consider it a hotfix. Previously, I saved the created file paths from self.train_filenames as lrs2.txt or lrs3.txt and reloaded them. It should be straightforward for you to adapt this approach for LRS2 experiments. However, since we need to accommodate various combinations—LRS2, LRS3, LRS2+LRS3, and LRS2+LRS3+VoxCeleb—it will take some time to finalize a comprehensive solution. Please feel free to make any necessary modifications on your end before I release a more complete update. Lastly, as you may have noticed, the code uses np.ceil(). The slight discrepancy—like handling 156 frames as 160 frames once or twice—shouldn’t significantly affect your results.