I have 1.5 million training inputs, which will take about 2 weeks to train. My concern is “what if the training fails after 10 or 13 days, towards the end of the training?”, resulting in a waste of time and money. So I have been trying to figure out how to train a model, save it, then continue tra…

Is iterative training advisable?

dblakely December 27, 2023, 3:02pm 2

Hi @capnchat, why not save checkpoints during training in case it crashes and then reload from the last checkpoint if needed?

2 Likes

Topic		Replies	Views
Training a model iteratively instead of all at once (RuntimeError: Expected all tensors to be on the same device, but found at least two devices...) Beginners	4	824	December 28, 2023
Retraining peft model Intermediate	3	2954	March 1, 2024
PEFT fine-tuning as slow as full model fine-tuning Intermediate	3	1614	December 6, 2023
Training models for smaller epochs and then continue trianing 🤗Transformers	5	1331	January 16, 2021
Autotrain Advanced (local) finished training between epochs i.e not sure it actually completed 🤗AutoTrain	2	1145	October 13, 2023