My model is trained with additional token MODEL_NAME = "mistralai/Mistral-7B-Instruct-v0.1" tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME) tokenizer.add_special_tokens({'pad_token': '[PAD]'}) model.resize_token_embeddings(len(tokenizer)) However, I am unable to load the model in the standar…

Unable to load a model with added special token

yangenius April 3, 2024, 5:08pm 2

Topic		Replies	Views
Loading pre-trained models with AddedTokens 🤗Transformers	2	303	October 14, 2024
Size mismatch error in PEFT fine tuned model 🤗Transformers	4	693	July 2, 2024
Loading trained model with new vocab Beginners	2	982	April 10, 2024
Mixtral training creates additional embedded token and head weights 🤗Transformers	0	74	June 13, 2024
Cannot load pretrained tokenizer from 'IlyaGusev/mbart_ru_sum_gazeta' model Models	0	333	April 21, 2021