Clarification on dual scale early fusion in terramind_v1_base

[philippinel]

Hello,

I am currently training a land cover classification model using terramind_v1_base. From the documentation and the paper, TerraMind supports a dual-scale training objective (pixel-level embeddings + unimodal tokens from the FSQ-VAEs).

However, I build my model as follow and I am a bit confused on what is going on. My understanding is that this only uses pixel-level patch embeddings for all modalities (including NDVI). The unimodal tokenizer branch (FSQ-VAE tokens) is not used automatically in this setup.

model = BACKBONE_REGISTRY.build(
    "terramind_v1_base",
    modalities=["S2L2A", {"fc": 3}, "DEM", {"NDVI": 1}],
    pretrained=True,
)

model = terratorch.tasks.ClassificationTask(
    model_factory="EncoderDecoderFactory",
    model_args={
        "backbone": "terramind_v1_base",
        "backbone_pretrained": True,
        "backbone_modalities": ["S2L2A", {"fc" : 3}, "DEM", {"NDVI" : 1}],
        "necks": [
            {"name": "SelectIndices", "indices": [2, 5, 8, 11]}, 
            {"name": "ReshapeTokensToImage", "remove_cls_token": False},  
        ],
        "decoder": "IdentityDecoder",
        "num_classes": 7,
        "head_dropout": 0.1,
    },
    loss="ce",
    optimizer="AdamW",
    lr=1e-4,
    optimizer_hparams={"weight_decay": 0.01},
    scheduler="ReduceLROnPlateau",
    scheduler_hparams={"factor": 0.5, "patience": 5},
    freeze_backbone=True,
    freeze_decoder=True,
    freeze_head=False,
    class_names=["Forest", "Grassland", "Cropland", "Buildup", "Baresoil", "Water", "Mangroves"],
)

Thus, my questions are:

1. Does terramind_v1_base handle only pixel embeddings?
2. If I want to leverage the dual-scale fusion (pixel + unimodal tokens) as described in the paper (which reportedly improves performance), how should I proceed in practice?

For example, would the right approach be:
Build a pixel-level backbone with ["S2L2A", "DEM", "FC"] and then separately encode NDVI (and other modalities if wanted) with FULL_MODEL_REGISTRY.build("terramind_v1_tokenizer_ndvi"),. Concatenate the tokens before feeding into the backbone (or at some neck stage). Train the fused model for classification.

[blumenstiel]

Hi @philippinel, you can use the NDVI tokeniser by just defining the modality as "NDVI" rather than as a dict. However, I would not recommend it as the model would use a fully ViT-b model just for encoding your NDVI data, basically doubling your parameter count.

So for you questions:

1. TM can also use tokeniser for encoding modalities. However, I would strongly recommend to not do this!
2. Use NDVI as modality. TerraTorch automatically adds the tokenizer to the TM model, you don't have to do anything else. If you want to use the tokenizer for other modalities that do have a pre-trained patch embedding, you have to specify the exact name that we used in pre-training (tok_sen1grd@224, tok_sen2l2a@224, tok_dem@224, etc.)

Overall, I would just use a new patch embedding as you currently do. Please note that you do have to train the full model, so please use:

    freeze_backbone=False,
    freeze_decoder=False,

[philippinel]

I am sorry but you're answer is not really clear to me.

1. Do you recommend to use terramind_v1_base with NDVI as a modality (as follow) ?

model = BACKBONE_REGISTRY.build(
    "terramind_v1_base",
    modalities=["S2L2A", {"fc": 3}, "DEM", "NDVI"],
    pretrained=True,
)

model

2. If I used the previous model, for existing modalities (s2, dem, NDVI), token level is implemented within the model, and so I do not need to call any unimodal tokeniser (terramind_v1_tokenizer_s2l2a, terramind_v1_tokenizer_dem, terramind_v1_tokenizer_ndvi). Only the new modality won't have a unimodal tokeniser. Thus I'll have pixel level + token level for modalities and only pixel-level for new modality. I am understanding it well?

3. Do not really understand this " If you want to use the tokenizer for other modalities that do have a pre-trained patch embedding, you have to specify the exact name that we used in pre-training (tok_sen1grd@224, tok_sen2l2a@224, tok_dem@224, etc.)"

4. is there a way to keep the encoder weights frozen? I don't see the point of retraining the whole model (goes against the idea of a 'Foundation Model' no??)