CLIP-like models do not support .add_adapter method

🤗Transformers
1

When working with LoRA, it is common to use .add_adapter method to supply a particular layers of various NLP models such as DeBERTa, FLAN, T5, etc., with adapter inside those selected layers.
This method differs from get_peft_model, because the first one do not create a wrapper outside basic the model which, sometimes, comes in handy.

The problem is that this .add_adapter method is not supported for CLIP, ViT and other popular vision models.

If I am wrong, please tell me.
Best regards.

2

Hi,

The following works for me:

from transformers import AutoModelForImageClassification
from peft import LoraConfig

config = LoraConfig(
    r=16,
    lora_alpha=16,
    target_modules=["query", "value"],
    lora_dropout=0.1,
    bias="none",
    modules_to_save=["classifier"],
)
model = AutoModelForImageClassification.from_pretrained("google/vit-base-patch16-224")

model.add_adapter(config)

for name, param in model.named_parameters():
    print(name, param.shape)

This is thanks to the PEFT integration in Transformers: PEFT integrations.

See also this tutorial: Image classification using LoRA.