Hi! I’ve been consulting this page: https://huggingface.co/transformers/parallelism.html#data-parallel that says using DDP with transformers is “almost trivial”. But is there an example available? Am I supposed to follow https://pytorch.org/tutorials/intermediate/ddp_tutorial.html just as if I…

Using Transformers with DistributedDataParallel — any examples?

sgugger October 15, 2021, 1:08pm 6

Then you just need to properly launch your training script, see here.

Topic		Replies	Views
Which data parallel does trainer use? DP or DDP? 🤗Transformers	2	6232	August 17, 2022
Transformer model parallel does not work with Pytorch DDP for multi-node training 🤗Transformers	0	513	September 1, 2022
Trainer API for Model Parallelism on Multiple GPUs 🤗Transformers	5	3981	September 10, 2024
Problem in training iterable dataset 🤗Datasets	1	1002	December 26, 2023
How to run an end to end example of distributed data parallel with hugging face's trainer api (ideally on a single node multiple gpus)? Intermediate	17	17463	September 6, 2023