I try to search in the doc. But I didn’t find the answer anywhere. Thank you

It depends if you launch your training script with python (in which case it will use DP) or python -m torch.distributed.launch (in which case it will use DDP).

Which data parallel does trainer use? DP or DDP?

🤗Transformers

John6666 August 30, 2025, 2:42am 5

Hmm… Have you tried launching it via accelerate or torchrun?

# single node, 2 GPUs
torchrun --nproc_per_node=2 train.py
# or
accelerate launch --num_processes=2 train.py

Accelerator selection

Topic		Replies	Views
Model's evaluation in DDP training is using only one GPU Beginners	1	1062	September 14, 2023
Using Transformers with DistributedDataParallel — any examples? Intermediate	11	23485	May 8, 2023
Can't use multi GPU in evaluation from Trainer 🤗Transformers	3	970	December 6, 2023
Running a Trainer in DistributedDataParallel mode 🤗Transformers	1	1450	October 24, 2020
Trainer is not using multiple GPUs in the DP setup Beginners	0	826	April 9, 2023

Which data parallel does trainer use? DP or DDP?

Accelerator selection

Related topics