when I test tokenizer.tokenize(‘how do you do’) （RobertaTokenizer in pytorch_transformers.tokenization_roberta.py ）, it returns [‘how’, ‘Ġ’, ‘do’, ‘Ġ’, ‘you’, ‘Ġ’, ‘do’], wants to know where is the wrong

Issue with tokenizer.tokenize

facehugger2020 November 16, 2020, 7:21pm 3

See also this post in the forum.

Topic		Replies	Views
RobertaTokenizer decode and tokenize do not have the same output 🤗Tokenizers	0	248	October 24, 2023
Punctuation and Spaces in RoBERTa Tokenizer for NER with Pre-tokenized Data 🤗Transformers	0	593	January 16, 2022
Build a RoBERTa tokenizer from scratch 🤗Tokenizers	5	3371	December 12, 2020
All my sequences get tokenized the same 🤗Tokenizers	2	614	February 12, 2022
Tokenizer decoding using BERT, RoBERTa, XLNet, GPT2 Beginners	7	8668	September 21, 2020