Inconsistent Metric Results in terramind_v1_base_sen1floods11.ipynb Testing

[eterna621]

Hi all! Thanks for the great work!

While executing terramind_v1_base_sen1floods11.ipynb in Google Colab, I observed inconsistencies in the resulting test metrics. This discrepancy appears when the fine-tuned model is tested in two different ways:

1. Passing the datamodule to the test function.

2. Passing the test_dataloader directly.

I've included screenshots. This likely explains why the predictions after the third epoch look incorrect and don't match the TensorBoard results.

Could you please investigate this metric discrepancy?

[blumenstiel]

Hi @eterna621, thanks for reporting. This looks like an issue from Lightning/TerraTorch. @Joao-L-S-Almeida have you seen this before?

It's interesting that test/loss is missing in the second run. Often this is the case if you run with half precision and NaNs appear somewhere in the model (or input data). Can you report your second run with full precision?

[eterna621]

Thank you for your quick response. Here is the second run with full precision:

I am not sure if this helps, but my hypothesis is there is an issue with the normalization. test_dataloader seems to be missing batch normalization.

[blumenstiel]

Thanks for testing! Agree, it must be something related to steps missing in the data loader that are applied in the datamodule. E.g., input normalization is handled not in transform but in another step in torchgeo/terratorch. This might also explain the accuracy close to 50%, which is basically random.

@Joao-L-S-Almeida Ideally we find a solution for this issue, or if possible we can raise an error or warning if a dataloader is passed directly.