NER model fine tuning with labeled spans


I’m looking to fine-tune an NER model (dslim/bert-base-NER-uncased) with my own data.

My annotations are of this form: for each example I have a piece of raw text (str) and a list of annotated spans of this form: {start_index: int, end_index: int, tag: str}

However, to fine-tune the NER model, I need to prepare X (tokens) and Y (token tags) for each example. So, those spans have to be translated into token tags, matching the model’s tokenizer.

Hope that makes sense.

Is there a way to handle this? Or, what would you recommend?


Hi folks! Does this make sense?


Yes that’s the right approach. You can take a look at Spacy’s offsets_to_biluo_tags method. It’s great to convert character index-level annotations to token annotations (in BILOU-format, which is a bit more exotic than IOB).

Thank you @nielsr!

The problem with that is that offsets_to_biluo_tags uses some spacy tokenizer right? And they use a different tokenizer from Huggingface BERT tokenizer for example. So don’t I need something that uses HF’s BERT tokenizer to go from spans to token labels?

you can convert the span to BIOS tags(other tag schema is similar) like this:

def convert_ents_to_bio(text, ents):
    tags = ["O"] * len(text)
    for ent in ents:
        start, end, label = ent["start"], ent["end"], ent["label"]
        if start == end:
            tags[start] = 'S-'+ label
            tags[start] = 'B-' + label
            tags[start+1: end + 1] = ['I-'+label]*(end - start)
    return tags

text = ["tom", "lives", "in", "Los", "Angeles"]
ents = [
    {"start":0, "end": 0, "label": "PERSON"},
    {"start":3, "end": 4, "label": "ADDRESS"},
convert_ents_to_bio(test, ents)