I am trying to finetune mT5 model on a single node with 8 Nvidia A100s. I load the model with datatype = torch.bfloat16 model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint, cache_dir=cache_dir,torch_dtype=datatype, device_map=“auto”) and use Seq2SeqTrainer num_gpus = torch.cuda.dev…

Starting the training using torchrun and specifying nproc_per_node fixed it. @muellerzr : A suggestion - Adding this to the ‘Using Trainer’ section in the blog From PyTorch DDP to Accelerate to Trainer would be helpful for others. torchrun --nproc_per_node 8 <trainingfile.py>

Model not copied to multiple GPUs when using DDP (using trainer)

🤗Accelerate

system Closed February 5, 2024, 11:33pm 3

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Seq2SeqTrainer multiple GPUs 🤗Transformers	2	115	January 22, 2025
Trainer is not using multiple GPUs in the DP setup Beginners	0	826	April 9, 2023
Code terminates without training while using accelerate 🤗Accelerate	3	183	April 13, 2024
Huggingface Seq2SeqTrainer uses accelerate so it cannot be run with DDP? 🤗Accelerate	1	563	January 24, 2024
How to run an end to end example of distributed data parallel with hugging face's trainer api (ideally on a single node multiple gpus)? Intermediate	17	18039	September 6, 2023

Model not copied to multiple GPUs when using DDP (using trainer)

Related topics