Hello friends,
When I finish my trainning i am using:
trainer.save_model("mode_name")
The line above saves 6 files, including a pytorch_model.bin that is in average 400mb in size. This is a problem to due to github size limitation to 100mb(I know i can use git LFS).
After saving my model I am loading it to use with these lines:
from transformers import pipeline, BertTokenizer
tokenizer = BertTokenizer.from_pretrained(path)
classifier = pipeline('text-classification',model=path, tokenizer=tokenizer, top_k=5)
So my trainning arguments are saving all epochs. I already know how to avoid this.
My question is: is there a way to load the model, remove all epochs but the best one ans save it again for deploy with a smaller .bin file? I dont know if am packing a lot of stuff that I dont need to use the model, that’s my concern.
Thanks a lot,