ValueError: The model did not return a loss from the inputs

Beginners
1

I am getting this error when I try to fine-tune a model. I have been trying to solve it for hours but I can’t figure out what is going on. Can somebody help?:

ValueError                                Traceback (most recent call last)
<ipython-input-129-3435b262f1ae> in <cell line: 1>()
----> 1 trainer.train()

3 frames
/usr/local/lib/python3.10/dist-packages/transformers/trainer.py in compute_loss(self, model, inputs, return_outputs)
   2763         else:
   2764             if isinstance(outputs, dict) and "loss" not in outputs:
-> 2765                 raise ValueError(
   2766                     "The model did not return a loss from the inputs, only the following keys: "
   2767                     f"{','.join(outputs.keys())}. For reference, the inputs it received are {','.join(inputs.keys())}."

ValueError: The model did not return a loss from the inputs, only the following keys: logits. For reference, the inputs it received are input_ids,attention_mask.

I also attach my code here!

from transformers import AutoModelForMaskedLM, AutoTokenizer, Trainer, TrainingArguments
import numpy as np
import evaluate

import torch
from datasets import load_dataset

dataset = load_dataset("xtreme", "PAN-X.eu", split='train')

print(dataset[0])

tokenizer = AutoTokenizer.from_pretrained("IParraMartin/EusBERTa", add_prefix_space=True)
model = AutoModelForMaskedLM.from_pretrained("IParraMartin/EusBERTa")

print(dataset[0])

def encode_data(examples):
    return tokenizer(examples['tokens'],
                     padding='max_length',
                     truncation=True,
                     max_length=512,
                     return_tensors="pt",
                     is_split_into_words=True)

encoded_dataset = dataset.map(encode_data, batched=True)

model.classifier = torch.nn.Linear(model.config.hidden_size, 6)

args = TrainingArguments(output_dir="./results",
                         num_train_epochs=3)

metric = evaluate.load('accuracy')

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

trainer = Trainer(model=model,
                  args=args,
                  train_dataset=encoded_dataset)

trainer.train()

Thank you!:hugs:

2

Hi,

For the model to return a loss, it should also be passed labels (i.e., the ground truth targets), besides the inputs (input_ids and attention_mask).

Refer to the masked language modeling (MLM) script for details: transformers/examples/pytorch/language-modeling/run_mlm.py at main · huggingface/transformers · GitHub.