Hi, I’m trying to load a pre-trained model from the local checkpoint. However, I get significantly different results when I evaluate the performance on the same validation set used in the training phase.

The code I’m using is an example notebook: “https://github.com/huggingface/peft/blob/main/examples/image_classification/image_classification_peft_lora.ipynb”

After the training, I save the model using the following:

trainer.save_model()

trainer.log_metrics(“train”, train_results.metrics)

trainer.save_metrics(“train”, train_results.metrics)

trainer.save_state()

Then, I load the model using:

new_model = AutoModelForImageClassification.from_pretrained(‘vit-base-patch16-224-in21k-finetuned-lora/checkpoint-18’)

new_trainer = Trainer(new_model, args,

eval_dataset=val_ds, tokenizer=image_processor,

compute_metrics=compute_metrics, data_collator=collate_fn)

new_trainer.evaluate(val_ds)

However, two trainers gave me completely different results:

{‘eval_loss’: 0.4082214832305908,

‘eval_accuracy’: 0.88,

‘eval_runtime’: 6.8795,

‘eval_samples_per_second’: 72.68,

‘eval_steps_per_second’: 0.581,

‘epoch’: 2.0}

and

{‘eval_loss’: 4.637523651123047,

‘eval_accuracy’: 0.016,

‘eval_runtime’: 6.0346,

‘eval_samples_per_second’: 82.856,

‘eval_steps_per_second’: 0.663}

Thus, it seems I’m not loading the model with random weights. Could you please tell me how to load the architecture and the weights?