Training time

[Ethereal24]

Hello, I would like to know approximately how long it would take to perform one round of fine-tuning on the GSM8K dataset. Could you provide some training information?

[kossisoroyce]

Training Time Estimates (1 epoch)

Model	Method	Hardware	Est. Time (1 epoch)
GPT-2 small (124M)	Full FT	single T4	~10–15 mins
GPT-2 medium	Full FT	A100	~5 mins
LLaMA-2 7B	LoRA	A100 40GB	~20–30 mins
Mistral 7B	QLoRA	1× A100	~30–40 mins
Phi-2	Full FT	A100	~15–20 mins
GPT-NeoX 20B	Full FT	4× A100	~1.5–2 hours
LLaMA-2 13B	LoRA	2× A100	~40–60 mins

[marawangamal]

How did you handle GPT2 without a chat template?

ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed!