Peformance metrics won't be calculated

[dkleine]

I am currently working on a project using an autoregressive model (LLama 3.2) for Named Entity Recognition (NER) by disabling the attention mask to make the model bidirectional (=full attention mask). This works so far pretty good, but I have issues calculating the metrics I have defined in calculate_metrics

This is the notebook: NER_decoder/NER-with-Llama.ipynb at d9f1dfef8e45a5a2e3d864c2c6aea4e70eee09d5 · d-kleine/NER_decoder · GitHub

As you can see, the loss will be calculated correctly, but the actual performance metrics (accuracy, recall, precision, F1) will not be displayed.

When predicting on the test set with predictions, labels, metrics = trainer.predict(test_dataset_tokenized):

• predictionsis tuple
• labels is empty (empty output)
• metrics is empty (empty output too, as labels is empty)

Do you have any idea why this happens and how to resolve this issue? I assume something is wrong with the calculate_metrics, tokenize_and_align_labels and data_collator - but I can be wrong about that.

[dkleine]

I just got it fixed and working. The task_type in the LoRA configuration was named wrongly (AutoModelForTokenClassification instead of TOKEN_CLS), without returning an error message that this task type is not even known.

I will raise an issue on that on GitHub as this should assert an error message.