Hugging Face Forums

Meanings of different brackets during tokenization

🤗Transformers

Harveenchadha December 10, 2021, 10:42pm 1

Hi,

What is the meaning of different brackets during tokenization ?

Like [ ], {} , ()

Topic		Replies	Views	Activity
Ġ token inserted by ByteLevelBPETokenizer 🤗Transformers	0	507	November 1, 2023
<unk> token in the output instead curly braces 🤗Tokenizers	0	494	March 25, 2023
What does 🤗 mean here and there in the document? Beginners	1	172	October 13, 2023
Is it possible to tokenize multiple text modalities? 🤗Transformers	1	451	September 1, 2022
T5 tokenizer vs t51.1 tokenizer 🤗Tokenizers	0	203	March 1, 2024