Deterministic execution

[lnicola]

This is probably more of a TerraTorch question, but is it possible to make the execution more deterministic? Looking at large_tile_generation.ipynb, I tried adding:

torch.manual_seed(42)
np.random.seed(42)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

but even when running on cpu, the LULC output is very different. I didn't check the raw probabilities or logits, though.

[lnicola]

CC @miquel-espinosa via #32 -- I have the reverse question, do you get the same outputs from the same seeds?

[miquel-espinosa]

We have observed very limited stochasticity from Terramind. Instead, it collapses to the conditional mean.

For instance, generating S2L2A solely conditioned on LULC will return very similar results for different seeds. But we know that the same LULC map could correspond to very different S2L2A images (e.g. seasonality changes, climatic zones, vegetation diversity).

Find here the link to our paper, in case you are interested in more details.
Paper: https://arxiv.org/abs/2603.03239
Website: https://miquel-espinosa.github.io/cop-gen/

[blumenstiel]

Hi @lnicola, probably some settings are missing, e.g. torch.cuda.manual_seed(seed). But I also don't know exactly what to set to make it reproducable 100%. Also I just saw that model.eval() is missing in the notebook, I am adding it now.

[blumenstiel]

Thanks @miquel-espinosa for sharing. You probably need to play with the generation settings in generate.py to increase diversity. But I cannot point you to a specific parameter to change.

[blumenstiel]

It is fixed in the notebook. I assume your differences are coming from the dropout layers in the model.

[lnicola]

Also I just saw that model.eval() is missing in the notebook

Yeah, I was actually wondering about that. Anyway, it didn't seem to make a difference.

probably some settings are missing, e.g. torch.cuda.manual_seed(seed)

I think torch.manual_seed handles that, see https://github.com/pytorch/pytorch/blob/v2.10.0/torch/random.py#L32-L64.

[blumenstiel]

You are right, the seed should be correct. Then I also don't know where this difference could come from. @jhnnsjkbk1 Any idea why the generation is not deterministic with fixed seeds?