Eval with trainer not running with PEFT LoRA model

🤗Transformers
1

Hi there,

I have a training script that when running with defined model and trainer, the evaluation loop works fine.
When wrapping the model with PEFT LoRA config evaluation doesn’t run. The training is completed properly but the evaluation loop every X steps just doesn’t seem to happen.
When I dug in a bit I saw it happened because for some reason in PEFT mode the trainer didn’t provide labels to the evaluation loop so it was skipped.
Here’s the package versions and script:

torch==2.0.1
transformers==4.32.1
peft==0.5.0

Original code:

class CLIPForSceneClassification(CLIPModel):

    def forward(self,
        pixel_values,
        labels,
        text_inputs):

        s = pixel_values.shape # B,S,3,H,W
        pixel_values = pixel_values.view([s[0]*s[1],s[2],s[3],s[4]])
        outputs = self.vision_model(pixel_values=pixel_values)
        outputs = outputs[1].view([s[0],s[1],-1])
        image_embeds = torch.mean(outputs,dim=1)

        text_outputs = self.text_model(**text_inputs)
        text_embeds = text_outputs[1]

        image_embeds = self.visual_projection(image_embeds)
        text_embeds = self.text_projection(text_embeds)

        # normalized features
        image_embeds = image_embeds / image_embeds.norm(p=2, dim=-1, keepdim=True)
        text_embeds = text_embeds / text_embeds.norm(p=2, dim=-1, keepdim=True)

        # cosine similarity as logits
        logit_scale = self.logit_scale.exp()
        logits = torch.matmul(image_embeds, text_embeds.t()) * logit_scale
        # logits_per_text = logits_per_image.t()

        loss = None
        if labels is not None:
            loss_fct = CrossEntropyLoss()
            loss = loss_fct(logits.view(-1, self.n_classes), labels.view(-1))

        return loss, logits

def compute_metrics(pred):
        labels = pred.label_ids
        preds = pred.predictions
        preds = preds.argmax(-1)
        precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average="weighted")
        acc = accuracy_score(labels, preds)
        return {"accuracy": acc, "f1": f1, "precision": precision, "recall": recall}

model = CLIPForSceneClassification.from_pretrained(args.model_name)

training_args = TrainingArguments(
    output_dir=args.model_dir,
    num_train_epochs=args.epochs,
    per_device_train_batch_size=args.train_batch_size,
    per_device_eval_batch_size=args.eval_batch_size,
    warmup_steps=args.warmup_steps,
    evaluation_strategy="steps",
    eval_steps=50,
    logging_dir=f"{args.output_data_dir}/logs",
    learning_rate=float(args.learning_rate),
    # label_names=["real", "animated"],
    report_to="none",
    use_mps_device=torch.backends.mps.is_available(),
    logging_steps=10,
    save_strategy="steps",
    save_steps=50,
    dataloader_num_workers=2,
    remove_unused_columns=False,
)

# create Trainer instance
trainer = Trainer(
    model=lora_model,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_dataset,
    eval_dataset=test_dataset,
)

Here’s what I modified with PEFT:

config = LoraConfig(
    r=4,
    lora_alpha=4,
    lora_dropout=0.0,
    target_modules=["visual_projection", "text_projection"],
    # modules_to_save=["seq.4"],
)
lora_model = get_peft_model(model, config)
lora_model.print_trainable_parameters()

...

# create Trainer instance
trainer = Trainer(
    model=lora_model,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_dataset,
    eval_dataset=test_dataset,
)

Would be happy for help here. Thanks!!!

1 Like
2

They haven’t made PEFT compatible with the find_labels function yet.

The easiest fix should be to add this argument in training args.

from transformers import TrainingArguments
training_args = TrainingArguments(...,label_names=['labels'])
6 Likes