How can I change the token id of a special token?

vitvit · January 6, 2022, 3:09pm

I am combining a tokenizer from model A with a pretrained model B.
I want to align special token ids.
I am able to change the token name such as but I cannot change the token id number.
How can I aproach this problem?

Topic		Replies	Views
Does tokenizer changed during model training Beginners	2	689	August 11, 2022
How to add special tokens to a pretrained model? Beginners	0	311	June 18, 2021
How to customize behavior of added special tokens in a pretrained tokenizer? Intermediate	0	414	May 5, 2021
Same sequence maps to different token ids Intermediate	0	331	August 29, 2022
How to add a new special token and initialize its embeddings to random values? Beginners	0	202	October 19, 2022

How can I change the token id of a special token?

Related Topics