[Feat] Frozen backbone finetuning

[Jaster1999]

Search before asking

• I have searched the RF-DETR issues and found no similar feature requests.

Description

The ability to set the backbone to be frozen for finetuning. something like freeze_backbone=True in model.train args e.g.

model.train(
dataset_dir="",
freeze_backbone=True,
)

Use case

Specifically this would be useful for training very large models on consumer gpu hardware but also to prevent overfitting on smaller datasets. The logic here being the object365 pretraining can create generalized features for object detection and we just adapt the object detection neck and head for the downstream task.

If it works we could potentially train more generalized models on small dataset (sacrificing better metrics for generalization)

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[isaacrob]

we decay the learning rate through the backbone already, you should be able to make this happen by setting lr_vit_layer_decay=0

[Jaster1999]

Ok cool, how does that rate work exactly as in it will multiply by the lr e.g. layer lr = lr_vit_layer_decay * lr?

[Jaster1999]

Thinking about this further, setting lr=0 wouldn't disable gradients for the backbone no? Ideally if we were effectively disabling the gradients via lr=0, we'd want the backbone to just not have gradients as it would be much faster to train and use less vram (e.g. if backbone was frozen in eval mode or disconnected from backprop).

[isaacrob]

True! Yeah wouldn't have those benefits. Prob worth adding feature then. Want to write pr?

[zhaoshuo1223]

没错！那样就没那些好处了。看来值得添加这个功能。想写个 PR 吗？

In main.py, I noticed that there seems to be a freeze_encoderparameter that will freeze DINOV2.

[VijayVignesh1]

I am not able to see the freeze_backbone option. I can raise a PR to add this feature.