Does tokenizer changed during model training

Beginners
1

I want to know during model training, does tokenizer alter it self, (i mean does it train) or it remain the same as original one.
For example i used the
AutoTokenizer.from_pretrained(model_name,local_files_only=False)
after model training, is there any change to tokenizer?

1 Like
2

Hi, Talha,

Tokenizer id does not change. For example, the token_id of [['sep']] is 102. It will be 102 through the training. But the embedding hidden state will be changed.

1 Like
3

One doubt. So lets say I have trained two models of “bert-base-uncased” to do different classification tasks(same number of classes).
So, now, can I use the tokenizer of one model1 to tokenize the other one? Will the result be different compared to using two different tokenizers?