Hugging Face Forums

Meanings of different brackets during tokenization

🤗Transformers

Harveenchadha December 10, 2021, 10:42pm 1

Hi,

What is the meaning of different brackets during tokenization ?

Like [ ], {} , ()

Topic		Replies	Views	Activity
Ġ token inserted by ByteLevelBPETokenizer 🤗Transformers	0	189	November 1, 2023
<unk> token in the output instead curly braces 🤗Tokenizers	0	404	March 25, 2023
Is it possible to tokenize multiple text modalities? 🤗Transformers	1	414	September 1, 2022
T5 tokenizer vs t51.1 tokenizer 🤗Tokenizers	0	125	March 1, 2024
What is a Text Pair in a tokenizer? Beginners	2	1664	July 18, 2023