Tokenization in a NER context

🤗Tokenizers
6

@lewtun - Just wanted to know, how to align multiple sub words into individual word in case of SentencePiece tokenizer like Albert, for NER?

For example,

My name is Abraham George and I am living in Philadelphia" .

Now this will be split into sub words, and predictions happens per sub word. Now how we align it back to the individual words.

@sgugger

2 Likes