Using Huggingface Trainer in Colab -> Disk Full

BramVanroy · May 3, 2021, 5:36pm

There are three default arguments that are relevant here, but seeing that you set save_total_limit=1 I am not sure what else could be being saved…

github.com

huggingface/transformers/blob/fe82b1bfa07aa054ef70583a561cb7c3978c697f/src/transformers/training_args.py#L161-L172


save_strategy (:obj:`str` or :class:`~transformers.trainer_utils.IntervalStrategy`, `optional`, defaults to :obj:`"steps"`):
    The checkpoint save strategy to adopt during training. Possible values are:


        * :obj:`"no"`: No save is done during training.
        * :obj:`"epoch"`: Save is done at the end of each epoch.
        * :obj:`"steps"`: Save is done every :obj:`save_steps`.


save_steps (:obj:`int`, `optional`, defaults to 500):
    Number of updates steps before two checkpoint saves if :obj:`save_strategy="steps"`.
save_total_limit (:obj:`int`, `optional`):
    If a value is passed, will limit the total amount of checkpoints. Deletes the older checkpoints in
    :obj:`output_dir`.

Can you see what’s actually on the disk?

Topic		Replies	Views
Saving checkpoints in drive 🤗Transformers	6	4078	July 19, 2022
"No space left on device" when using HuggingFace + SageMaker Amazon SageMaker	39	25646	October 10, 2023
Colab session crashing after using all available RAM Beginners	0	2421	January 16, 2021
Trainer API not pushing checkpoints to HUB 🤗Transformers	0	307	December 7, 2022
HuggingFace Trainer() does nothing - only on Vertex AI workbench, works on colab 🤗Transformers	2	1865	September 5, 2022

Using Huggingface Trainer in Colab -> Disk Full

Related topics