training error

[Zking668]

Hello, I trained the model according to the steps of the tutorial and specified 10 epochs, but the terminal outputs Engine run starting with ‘max_epochs=1 at each epoch’. as shown in https://github.com/aritraghsh09/GaMPEN/blob/master/tutorials/Training_and_Inference_tutorial.ipynb. And in the logging of Metrics in mlflow, I noticed that the train and test losses are only logged for two points, so is there a problem with the logic involved?

[aritraghsh09]

Hi @Zking668, thank you for notifying us of this bug!

I see that in https://github.com/aritraghsh09/GaMPEN/blob/master/tutorials/Training_and_Inference_tutorial.ipynb ; when we do training the printed outputs are similar -- the training does happen for 20 epochs but at the beginning of each epoch you see the output Engine run starting with max_epochs=1

I will try to look a bit more into this to see why PyTorch prints the Engine run starting with max_epochs=1 message at every epoch. But this doesn't affect the underlying training, as you can also see in the demo notebook. The training should go on for as many epochs as you specified.

In the terminal screenshot you posted, it seems the training ran for 5 epochs; what parameters did you pass to train.py to get this? If you specified epochs=5; then from the above output, your model did indeed finish training properly.

Regarding your MLFLow screenshot, we log training and testing metrics to mlflow only once at the beginning and once at the end; however if you look at the devel metrics you should be able to see numbers at the end of every epoch.

I will try to look more into this over the next two weeks; but if you figure out something on your own; feel free to post it here or open a PR.