My model is trained with additional token MODEL_NAME = "mistralai/Mistral-7B-Instruct-v0.1" tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME) tokenizer.add_special_tokens({'pad_token': '[PAD]'}) model.resize_token_embeddings(len(tokenizer)) However, I am unable to load the model in the standar…

Unable to load a model with added special token

yangenius April 3, 2024, 5:08pm 2

Topic		Replies	Views
Loading trained model with new vocab Beginners	2	828	April 10, 2024
Cannot load pretrained tokenizer from 'IlyaGusev/mbart_ru_sum_gazeta' model Models	0	299	April 21, 2021
Loading Peft model from checkpoint leading into size missmatch 🤗Transformers	6	2166	February 7, 2024
Loading opus-mt-es-en fails Models	0	688	September 10, 2021
Using loaded model with accelerate for inference 🤗Accelerate	3	1696	November 4, 2022