I tried to fine-tuned language model using the following code
The issue I face is that the model is saved using (trainer.save_mode()), but I don’t know how save tokenizer’s results as well?
I’ve tried to add tokenizer=tokenizer to trainer, but it ariesed error.
any help is really appreciated.
As the tokenizer is AutoTokenizer, you can use tokenizer.save_pretrained(your path) to save it.