When using the gpt2 tokenizer, I get output where each word starts with the Ġ. I’m not sure what the Ġ is? Any suggestions?
from transformers import GPT2TokenizerFast
tokenizer = GPT2TokenizerFast.from_pretrained('gpt2', add_prefix_space=True)
inputs = tokenizer("today is a nice day")
inputs.tokens()
['Ġtoday', 'Ġis', 'Ġa', 'Ġnice', 'Ġday']