Hugging Face Forums

Meanings of different brackets during tokenization

🤗Transformers

Harveenchadha December 10, 2021, 10:42pm 1

Hi,

What is the meaning of different brackets during tokenization ?

Like [ ], {} , ()

Topic		Replies	Views	Activity
Is it possible to tokenize multiple text modalities? 🤗Transformers	1	462	September 1, 2022
Issue with tokenizer.tokenize 🤗Tokenizers	3	504	November 16, 2020
Continuation token in pertained tokenizer bert-base-chinese 🤗Tokenizers	0	528	July 11, 2020
About the Tokenizers category 🤗Tokenizers	1	314	July 7, 2020
Ġ token inserted by ByteLevelBPETokenizer 🤗Transformers	0	569	November 1, 2023