Error when fine-tuning with the Trainer API

🤗Datasets
1

Hi,

I’m fine-tuning a model with the Trainer API and following these instructions: https://huggingface.co/docs/transformers/training#finetuning-in-pytorch-with-the-trainer-api

However, after I have defined the compute_metrics function and tried to run the script, it gave me the following error:

Traceback (most recent call last):
  File "/home/le/torch_tutorial/lm_1_perpl.py", line 77, in <module>
    trainer.train()
  File "/home/le/torch_tutorial/venv/lib/python3.9/site-packages/transformers/trainer.py", line 1391, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/home/le/torch_tutorial/venv/lib/python3.9/site-packages/transformers/trainer.py", line 1491, in _maybe_log_save_evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/home/le/torch_tutorial/venv/lib/python3.9/site-packages/transformers/trainer.py", line 2113, in evaluate
    output = eval_loop(
  File "/home/le/torch_tutorial/venv/lib/python3.9/site-packages/transformers/trainer.py", line 2354, in evaluation_loop
    metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
  File "/home/le/torch_tutorial/lm_1_perpl.py", line 67, in compute_metrics
    return metric.compute(predictions=predictions, references=labels)
  File "/home/le/torch_tutorial/venv/lib/python3.9/site-packages/datasets/metric.py", line 393, in compute
    self.add_batch(predictions=predictions, references=references)
  File "/home/le/torch_tutorial/venv/lib/python3.9/site-packages/datasets/metric.py", line 434, in add_batch
    batch = self.info.features.encode_batch(batch)
  File "/home/le/torch_tutorial/venv/lib/python3.9/site-packages/datasets/features/features.py", line 1049, in encode_batch
    encoded_batch[key] = [encode_nested_example(self[key], obj) for obj in column]
  File "/home/le/torch_tutorial/venv/lib/python3.9/site-packages/datasets/features/features.py", line 1049, in <listcomp>
    encoded_batch[key] = [encode_nested_example(self[key], obj) for obj in column]
  File "/home/le/torch_tutorial/venv/lib/python3.9/site-packages/datasets/features/features.py", line 853, in encode_nested_example
    return schema.encode_example(obj)
  File "/home/le/torch_tutorial/venv/lib/python3.9/site-packages/datasets/features/features.py", line 297, in encode_example
    return int(value)
TypeError: only size-1 arrays can be converted to Python scalars

Do you have any ideas on what can be causing it? I have not changed anything in my code except for adding the compute_metrics function (like in the tutorial) and adding the compute_metrics argument in Trainer (before this addition everything was working perfectly):

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=lm_datasets["train"],
    eval_dataset=lm_datasets["validation"],
    compute_metrics=compute_metrics,
)
2

What I would do in this case is simply add a print statement within the compute_metrics function to view your logits and labels, their shapes, et.

1 Like
3

I did print the shapes of the variables inside of compute_metrics but they seem to be fine (at least they have the same shape):

Shape logits:  (148, 128, 50265)
Shape labels: (148, 128)
Shape predictions:  (148, 128)

Or is it something off here?

4

You didn’t tell us which metric you are using so it’s hard to help you without that info. It might expect something flat for instance (if it’s accuracy).

1 Like
5

I was trying to use accuracy indeed but did not know that it expects flat values, so thanks!