I am training a DistilBert pretrained model for sequence classification with a pretrained tokenizer. I currently save the model like this: > model.save_pretrained(dir) > tokenizer.save_pretrained(dir) And load like this: > model.from_pretrained(dir) > tokenizer.from_pretrained(dir)). Weirdly …

Simple Save/Load of tokenizer not working

abhi November 4, 2020, 12:03am 3

Great, thanks!

Topic		Replies	Views
Issue in loading the saved tokenizer 🤗Tokenizers	1	239	January 4, 2024
How to save a fast tokenizer using the transformer library and then load it using Tokenizers? 🤗Tokenizers	7	3480	December 14, 2022
How to save my tokenizer using save_pretrained? Beginners	5	29545	August 13, 2021
Trained a tokenizer from scratch but problem when loading 🤗Transformers	0	484	October 8, 2023
Can't load tokenizer using from_pretrained Beginners	0	241	July 11, 2023