Loading Weights from Customized Model

🤗Transformers
1

Hello!

I have trained a ViTAdapter model, that I added a prediction head to, with the following code:

model = AutoAdapterModel.from_pretrained(
    'google/vit-base-patch16-224-in21k',
    from_tf=bool(".ckpt" in 'google/vit-base-patch16-224-in21k'),
    config=config,
    cache_dir=None,
    revision='main',
    use_auth_token=None,
    ignore_mismatched_sizes=False,
)

model.add_image_classification_head(
    "classification_head",
    num_labels = len(labels),
    id2label=id2label,
)

I now want to load up the weights, so I can use the model on new images. However, when I call the following:

model = AutoAdapterModel.from_pretrained("path_to_model_weights")

I get the error

Some weights of the model checkpoint at vit_adapter_orange_surgical_task/vit-adapter-surgery were not used when initializing ViTAdapterModel

I believe this is because the weights from the added prediction head cannot be initialized onto a basic ViTAdapterModel. Is there a way to call .from_pretrained from a custom model instead of AutoAdapterModel? Alternatively, could a prediction head be added onto the loaded model above and filled with the weights from the trained head?

I appreciate any help and advice!
Thank you