Load model from checkpoints occurs degraded performance

Beginners
1

Hi, I’m trying to load a pre-trained model from the local checkpoint. However, I get significantly different results when I evaluate the performance on the same validation set used in the training phase.

The code I’m using is an example notebook: “https://github.com/huggingface/peft/blob/main/examples/image_classification/image_classification_peft_lora.ipynb”

After the training, I save the model using the following:

trainer.save_model()
trainer.log_metrics(“train”, train_results.metrics)
trainer.save_metrics(“train”, train_results.metrics)
trainer.save_state()

Then, I load the model using:

new_model = AutoModelForImageClassification.from_pretrained(‘vit-base-patch16-224-in21k-finetuned-lora/checkpoint-18’)
new_trainer = Trainer(new_model, args,
eval_dataset=val_ds, tokenizer=image_processor,
compute_metrics=compute_metrics, data_collator=collate_fn)
new_trainer.evaluate(val_ds)

However, two trainers gave me completely different results:
{‘eval_loss’: 0.4082214832305908,
‘eval_accuracy’: 0.88,
‘eval_runtime’: 6.8795,
‘eval_samples_per_second’: 72.68,
‘eval_steps_per_second’: 0.581,
‘epoch’: 2.0}

and

{‘eval_loss’: 4.637523651123047,
‘eval_accuracy’: 0.016,
‘eval_runtime’: 6.0346,
‘eval_samples_per_second’: 82.856,
‘eval_steps_per_second’: 0.663}

Thus, it seems I’m not loading the model with random weights. Could you please tell me how to load the architecture and the weights?

2

Ahh, My problem. I was using LoRA thus the new model can not load the corresponding weights.

3

For anyone wanna know how to load a pre-trained PEFT model, here is an example:

new_model = AutoModelForImageClassification.from_pretrained(
‘“google/vit-base-patch16-224-in21k”’,
label2id=label2id,
id2label=id2label,
ignore_mismatched_sizes=True,
)
lora_model = get_peft_model(new_model, config)
lora_model = lora_model.from_pretrained(new_model,
model_id=‘vit-base-patch16-224-in21k-finetuned-lora/checkpoint-18’)