Do we need to explicity save the model if the save steps is not a multiple of the num steps with HF?

[brando]

Do we need to explicity save the model if the save steps is not a multiple of the num steps with HF trainera?

trainer.train()
    trainer.save_model(output_dir=output_dir)  # TODO is this relaly needed?

cross: Discord
ref: Do we need to explicity save the model if the save steps is not a multiple of the num steps with HF?

[brando]

Do we need to explicitly save a Hugging Face (HF) model trained with HF trainer after the trainer.train() even if we are checkpointing?

related question: huggingface transformers - Do we need to explicitly save a Hugging Face (HF) model trained with HF trainer after the trainer.train() even if we are checkpointing? - Stack Overflow

[brando]

going to use:

    # - Make sure to save best checkpoint TODO: do we really need this? https://stackoverflow.com/questions/77261009/do-we-need-to-explicitly-save-a-hugging-face-hf-model-trained-with-hf-trainer
    final_ckpt_dir = output_dir / f'ckpt-{max_steps}'
    final_ckpt_dir.mkdir(parents=True, exist_ok=True)
    trainer.save_model(output_dir=final_ckpt_dir)  # TODO is this relaly needed? https://discuss.huggingface.co/t/do-we-need-to-explicity-save-the-model-if-the-save-steps-is-not-a-multiple-of-the-num-steps-with-hf/56745
    print('Done!\a')

[brando]

if you want to save tokenizer I think you need to do:

tokenizer.save_pretrained(training_args.output_dir)

e.g.,

    # note: seems trainer doesn't save tokenizer automatically https://chat.openai.com/c/c40db3a8-b614-40e0-b492-67319a1807e7 
    trainer.save_model(output_dir=output_dir)  # TODO is this really needed? https://discuss.huggingface.co/t/do-we-need-to-explicity-save-the-model-if-the-save-steps-is-not-a-multiple-of-the-num-steps-with-hf/56745
    ## tokenizer.save_pretrained(output_dir=output_dir)  # ref: https://discuss.huggingface.co/t/do-we-need-to-explicity-save-the-model-if-the-save-steps-is-not-a-multiple-of-the-num-steps-with-hf/56745/3

final ref: huggingface transformers - Do we need to explicitly save a Hugging Face (HF) model trained with HF trainer after the trainer.train() even if we are checkpointing? - Stack Overflow