NER - aggregation_strategy

I have custom trained a distilBERT NER model for extracting skills from Job Descriptions.

I am using aggregation_strategy = first.

There are my model results

The issues I am facing :

  1. The entity extraction fails, if the input text contains entities with space i.e., if we input ‘machine learning’ instead of ‘machinelearning’, the model ends up extracting only machine as a skill.

  2. Aggregation is grouping phrases in the sentence together.
    For example :

image

These issues persist with other aggregation strategies also i.e., simple/average/max

Any thoughts on why this is happening, and how to resolve this.

Thanks

Hey im working on the same project and it would help me greatly if you could lend me a hand on how you did it. Im kinda lost :smiley: