Based on the CTRL approach on GPT2, i’m trying to add tokens in order to control my text generation style. Is there a difference between adding a token as a regular one and adding it as a special token?

Regular tokens vs special tokens

Mahdip72 January 8, 2024, 5:30am 6

@lewtun Could you help us in this?

We can not extend the embedding layer by adding new tokens using ESM-2 models.

Topic		Replies	Views
How to customize behavior of added special tokens in a pretrained tokenizer? Intermediate	0	557	May 5, 2021
Slow Tokenizer adds whitespace after special token 🤗Transformers	4	1026	August 8, 2023
Why we need to add special tokens to tasks other than classification? 🤗Tokenizers	0	846	November 17, 2021
Preprocessing data for custom tokenizer 🤗Transformers	0	246	October 21, 2022
Is_split_into_words in encode_plus Beginners	0	271	February 27, 2023