Saving tokenizer's configuration

Hi,

I tried to fine-tuned language model using the following code

The issue I face is that the model is saved using (trainer.save_mode()), but I don’t know how save tokenizer’s results as well?

I’ve tried to add tokenizer=tokenizer to trainer, but it ariesed error.
any help is really appreciated.

hi,

As the tokenizer is AutoTokenizer, you can use tokenizer.save_pretrained(your path) to save it.