Issue with tokenizer.tokenize

🤗Tokenizers
1

when I test tokenizer.tokenize(‘how do you do’) （RobertaTokenizer in pytorch_transformers.tokenization_roberta.py ）, it returns [‘how’, ‘Ġ’, ‘do’, ‘Ġ’, ‘you’, ‘Ġ’, ‘do’], wants to know where is the wrong

2

There is some discussion in this therad and this. Perhaps it helps?

3

See also this post in the forum.

4

where is the wrong

That’s a new one. I haven’t seen that expression before. LOL