Load model from checkpoints occurs degraded performance

Hi, I’m trying to load a pre-trained model from the local checkpoint. However, I get significantly different results when I evaluate the performance on the same validation set used in the training phase.

The code I’m using is an example notebook: “https://github.com/huggingface/peft/blob/main/examples/image_classification/image_classification_peft_lora.ipynb

After the training, I save the model using the following:

trainer.log_metrics(“train”, train_results.metrics)
trainer.save_metrics(“train”, train_results.metrics)

Then, I load the model using:

new_model = AutoModelForImageClassification.from_pretrained(‘vit-base-patch16-224-in21k-finetuned-lora/checkpoint-18’)
new_trainer = Trainer(new_model, args,
eval_dataset=val_ds, tokenizer=image_processor,
compute_metrics=compute_metrics, data_collator=collate_fn)

However, two trainers gave me completely different results:
{‘eval_loss’: 0.4082214832305908,
‘eval_accuracy’: 0.88,
‘eval_runtime’: 6.8795,
‘eval_samples_per_second’: 72.68,
‘eval_steps_per_second’: 0.581,
‘epoch’: 2.0}


{‘eval_loss’: 4.637523651123047,
‘eval_accuracy’: 0.016,
‘eval_runtime’: 6.0346,
‘eval_samples_per_second’: 82.856,
‘eval_steps_per_second’: 0.663}

Thus, it seems I’m not loading the model with random weights. Could you please tell me how to load the architecture and the weights?

Ahh, My problem. I was using LoRA thus the new model can not load the corresponding weights.

For anyone wanna know how to load a pre-trained PEFT model, here is an example:

new_model = AutoModelForImageClassification.from_pretrained(
lora_model = get_peft_model(new_model, config)
lora_model = lora_model.from_pretrained(new_model,