Hugging Face Forums
Meanings of different brackets during tokenization
🤗Transformers
Harveenchadha
December 10, 2021, 10:42pm
1
Hi,
What is the meaning of different brackets during tokenization ?
Like [ ], {} , ()
Related topics
Topic
Replies
Views
Activity
Ä token inserted by ByteLevelBPETokenizer
🤗Transformers
0
507
November 1, 2023
<unk> token in the output instead curly braces
🤗Tokenizers
0
494
March 25, 2023
What does 🤗 mean here and there in the document?
Beginners
1
172
October 13, 2023
Is it possible to tokenize multiple text modalities?
🤗Transformers
1
451
September 1, 2022
T5 tokenizer vs t51.1 tokenizer
🤗Tokenizers
0
203
March 1, 2024