QAT for LLM

[SingJumpRapp]

Is there some documents about the QAT code for LLM? I want to do the QAT on llama3 but I can only find the code for mobilenet_v2, I think there is some gaps between different type models, could you provider me some help?

[lifelongeeek]

There seems qat script for llama2-7B.
https://github.com/quic/aimet/blob/develop/Examples/torch/quantization/llm_qat_kd/finetune_llm_qat_kd.py

I wonder how much quantization loss decreased with this recipe.

[quic-bhushans]

Hi SingJumpRapp what issues are you facing for QAT on LLM? current infra should allow you to run it through

please re-open issue if you still face any issues