You need to launch with torchrun --n_procs_per_node=NGPUS run_clm.py ... in order to enable multi-GPU. Another option is to use Accelerate’s CLI launcher directly:
accelerate launch --multi_gpu --num_processes=NGPUS run_clm.py...
2 Likes