Ask for help with prediction results of Named Entity Recognition Task

🤗Transformers
4

It looks like you are not using the “fast” version of the tokenizer. Check to make sure.

https://huggingface.co/transformers/model_doc/roberta.html#robertatokenizerfast
from transformers import RobertaTokenizerFast
tokenizer = RobertaTokenizerFast.from_pretrained("roberta-base")

tokenizer(“Hello world”)[‘input_ids’]
[0, 31414, 232, 328, 2]
tokenizer(" Hello world")[‘input_ids’]
[0, 20920, 232, 2]