Online training options

Thibaut · February 28, 2025, 7:20pm

Hi,

I’m not sure to understand correctly the options to train a model online with Hugging Face.

The most obvious option is AutoTrain. However this is not an option for us as Instance Segmentation is not supported.

I’ve got suggestions about using spaces. However in the documentation this does not seems to be something officially supported. Maybe more a hack?

I’ve also got suggestions about using the Inference API. It does not seems intuitive as it is called “Inference” but, after all, it supports GPU.

The most obvious option I see in the documentation would be to use Amazon Sagemaker integration. It’s a bit of overload but it maybe a reliable option.

What is your process in such cases ?

Regards

John6666 · March 1, 2025, 6:24am

If you want to do it for free, you can use Hugging Face’s CPU space, but I think the more common method is to use Google Colab Free. The Hugging Face course also uses that method…

Thibaut · March 2, 2025, 3:14pm

Thanks, we have enterprise HF and we are ok to spend money. Also, the training time will exceed Google Collab max up time. So I guess the solution is to use Amazon Sagemaker integration.

John6666 · March 2, 2025, 8:04pm

Then there are a lot of options. Trainer works in most environments as long as you have Python 3.10 and PyTorch, so I think you can run it on most cloud services other than SageMaker.
For example, you could create a dummy GUI in Hugging Face Spaces and use that to train. With general services, you don’t even need a GUI.

Well, when it comes to stability, I’m not sure. Trainer has an option to save checkpoints frequently, so if it crashes, it might be quicker to devise a way to resume from there.

Resume

github.com/huggingface/transformers

how to continue training from a checkpoint with Trainer?

opened 03:24AM - 17 Sep 20 UTC

closed 08:19PM - 20 Sep 20 UTC

fumpe

# ❓ Questions & Help ## Details I am trying to continue training my model …(gpt-2) from a checkpoint, using Trainer. However when I try to do it the model starts training from 0, not from the checkpoint. I share my code because I don't know where I'm making the mistake. ``` import torch device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') from transformers import TextDataset,DataCollatorForLanguageModeling, AutoTokenizer, GPT2LMHeadModel, Trainer, TrainingArguments tokenizer = AutoTokenizer.from_pretrained("gpt2-large") train_dataset = TextDataset( tokenizer=tokenizer, file_path='textfile (1).txt', block_size=128) data_collator = DataCollatorForLanguageModeling( tokenizer=tokenizer, mlm=False, ) model = GPT2LMHeadModel.from_pretrained("checkpoint-9500").to(device) ##HERE I LOAD FROM CHECKPOINT training_args = TrainingArguments( output_dir='./results', # output directory num_train_epochs=4, # total # of training epochs per_device_train_batch_size=1, # batch size per device during training per_device_eval_batch_size=64, # batch size for evaluation warmup_steps=500, # number of warmup steps for learning rate scheduler weight_decay=0.01, # strength of weight decay logging_dir='./logs', # directory for storing logs ) trainer = Trainer( model=model, args=training_args, data_collator=data_collator, train_dataset=train_dataset, #eval_dataset=validation_dataset, prediction_loss_only=True, ) trainer.train() ``` Thanks a lot for the help.

For cost efficient training

Thibaut · March 3, 2025, 1:18pm

Thanks a lot, we will try integration with SageMaker

Topic		Replies	Views
How to train hugging face model? 🤗Transformers	0	332	April 14, 2023
Non-Coder Training Question Beginners	5	346	February 20, 2025
Train a model with autotrain on huggingface using the API 🤗AutoTrain	4	274	May 30, 2024
Training on Sagemaker with Trainer() Instance Amazon SageMaker	6	2263	November 3, 2021
Online learning in a 🤗 Space Intermediate	2	625	December 1, 2021

Online training options

Resume

For cost efficient training

Related topics