when I test tokenizer.tokenize(‘how do you do’) （RobertaTokenizer in pytorch_transformers.tokenization_roberta.py ）, it returns [‘how’, ‘Ġ’, ‘do’, ‘Ġ’, ‘you’, ‘Ġ’, ‘do’], wants to know where is the wrong

Issue with tokenizer.tokenize

facehugger2020 November 16, 2020, 7:21pm 3

See also this post in the forum.

Topic		Replies	Views
RobertaTokenizer decode and tokenize do not have the same output 🤗Tokenizers	0	247	October 24, 2023
Punctuation and Spaces in RoBERTa Tokenizer for NER with Pre-tokenized Data 🤗Transformers	0	586	January 16, 2022
Using roberta for token-classification, strange characters Models	0	268	July 10, 2023
All my sequences get tokenized the same 🤗Tokenizers	2	609	February 12, 2022
Build a RoBERTa tokenizer from scratch 🤗Tokenizers	5	3355	December 12, 2020