Different results from checkpoint evaluation when loading fine-tuned LLM model

[botkop]

probably related:

Llama-2 Sequence Classification: Much lower accuracy on inference from checkpoint compared to model

I’m finetuning a Llama-2 sequence classification model with peft and qlora, and evaluating every 100 steps. I also save a checkpoint every 100 steps. When I load the checkpoint and do inference on the same validation set as during training, the accuracy is really much lower. Here’s the relevant code: Training: q_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16 ) model = AutoModelForSeq…