when I test tokenizer.tokenize(‘how do you do’) （RobertaTokenizer in pytorch_transformers.tokenization_roberta.py ）, it returns [‘how’, ‘Ġ’, ‘do’, ‘Ġ’, ‘you’, ‘Ġ’, ‘do’], wants to know where is the wrong

Issue with tokenizer.tokenize

facehugger2020 November 16, 2020, 7:21pm 3

See also this post in the forum.

Topic		Replies	Views
Tokenizer decoding using BERT, RoBERTa, XLNet, GPT2 Beginners	7	5131	September 21, 2020
RobertaTokenizer decode and tokenize do not have the same output 🤗Tokenizers	0	171	October 24, 2023
Using roberta for token-classification, strange characters Models	0	173	July 10, 2023
RoBERTa Tokenizer Java Implementation 🤗Tokenizers	1	1824	November 29, 2022
Training RoBERTa from scratch: error? 🤗Transformers	0	476	August 26, 2021