I’m fine-tuning GPT-2 on a lot of data and ended up using all of my disk space on google drive. I’m not sure which args prevent too many checkpoints. Right now I’m getting a new checkpoint at every 500 example, but I’d like to avoid making so many checkpoints. At the very least only keep the best ch…

How to prevent too many checkpoints with run_clm.py?

yubi-sanprit November 7, 2022, 7:59am 5

@sgugger @BramVanroy @JacquesThibs How can we implement lazy loading of data in RAM while training the model from scratch?

Topic		Replies	Views
Checkpoints and disk storage 🤗Transformers	15	8115	June 2, 2024
Saving only the best performing checkpoint 🤗Transformers	19	18239	May 23, 2023
Question Regarding trainer arguments:: load_best_model_at_end Beginners	2	1958	April 19, 2021
Tainer.train() - save_total_limit - latest vs best Beginners	0	413	June 2, 2022
Save only best model in Trainer 🤗Transformers	31	85810	June 25, 2024