Extension of the question discussed in #1943

[rithika2035]

Hi @felixdittrich92,

So I was going through all the chats in discussion #1943, and I did follow all the steps and I am able to get the result as intended. But adding the "₹" symbol in the vocab and then training it, just messed up the confidence scores. So when I ran inference for an invoice image the following were the issues I faced:

1. It was correctly recognizing some words but again the confidence given were low.
2. Some words were giving spelling mistakes (when I tested the pretrained model it didn't make mistakes, and confidence were pretty high).
3. If there is noise in the image, or for example the image has a qr code, the original pretrained model doesn't recognize it as a word as give some characters like "O" or "|". But the one that I trained gives some random symbols including "₹".

So is there any way to fix this?

[felixdittrich92]

Hi @rithika2035 👋

The issue is still that the pretrained models tend to "forget" very quickly. Training for fewer epochs (1–3) and possibly freezing some layers can help. There are two common strategies: either freeze only the backbone, or freeze everything up to the head - freezing up to the head depends on the architecture you want to fine tune.

Ideally, you'd combine a subset of the pretraining dataset with your new data. Unfortunately, that's not possible in this case, as the original dataset is a private, internal Mindee resource.

As for point 3 - that sounds more like a detection issue. QR codes shouldn’t be detected as text, so fine-tuning the detection model on a few targeted samples might help.

Best regards,
Felix

[rithika2035]

Hi @felixdittrich92,
So for crrn_vgg16_bn arch, I freezed everything up to the head and did the training. And I figured that I'll be needing the pretraining dataset to get :')
So in future will DocTR release model weights which is trained for recognize symbols like "₹"?
For the project I am doing I need the confidence scores as well, but after training it gives me low confidence scores even for correct words, so then it is difficult to determine the confidence of the outputs.