Hi all, I had a question regarding making changes to the tokenizer’s vocabulary. I want to register a new token’s id in the vocab, but I don’t want to add a new token and expand the size of the vocab.
Specifically, assume we are working with tokenizer from bert-base-cased. Now I want to register “qazwsx” to id 1, which is occupied by “[unused1]”. So I want to replace “[unused1]” : 1 by “qazwsx” : 1. Do you know how to achieve this?
Another question is how to synchronize two tokens with the same id, instead of overwrite it as above. For example, instead of replacing “[unused1]” : 1 by “qazwsx” : 1, I want to keep both in the new vocab, so that both of them will be tokenized as 1.
Thank you so much for the help!