NER model only predicts the outside 'O' tag

[CDobbs]

Hi

I’ve managed to produce an NER module following this tutorial. Although I am using the ScienceIE dataset (Process, Material, Task entities).

The issue is that my model only predicts the ‘O’ tag… I can understand that the model does it because the majority of tags are ‘O’ but obviously this isn’t solving the task.

I’ve created a custom compute_metrics function that calculates f1 score for each entity and the macro-average. However, this doesn’t help steer the model. Do I need to create a custom loss function?

I’ve tried googling around for this issue but haven’t had much luck. Sorry if this is a repeat issue.

Thanks for your help.

[lewtun]

hey @CDobbs

@CDobbs

I’ve created a custom compute_metrics function that calculates f1 score for each entity and the macro-average. However, this doesn’t help steer the model. Do I need to create a custom loss function?

if you’re using the Trainer for fine-tuning, then you shouldn’t need a custom loss function. speaking of losses, does the train / validation loss decrease during fine-tuning?

without seeing more details / code on your approach, it’s hard to debug the problem - can you share them?

ps. you might find it instructive to see the “official” tutorial on NER which contains many more details than the one you linked to: Google Colab