Using deepspeed script launcher vs accelerate script launcher for TRL

🤗Accelerate
1

I’ve been trying to figure out the nature of the deepspeed integration, especially with respect to huggingface accelerate.

It seems that the trainer uses accelerate to facilitate deepspeed. But when I look at the documentation, it seems that we still use deepspeed as the launcher, or the pytorch distribute

deepspeed --num_gpus=2 your_program.py <normal cl args> --deepspeed ds_config.json

or

python -m torch.distributed.launch --nproc_per_node=2 your_program.py <normal cl args>

But it didn’t mention using the accelerate launcher. I’m confused by this since trainer uses accelerate to facilitate the deepspeed integration.

As for the TRL library, it seems that it’s using the accelerate library for it’s trainers as well, but for the TRL library, the official way to launch it is to use the accelerate launcher

accelerate launch --config_file=examples/accelerate_configs/deepspeed_zero{1,2,3}.yaml --num_processes {NUM_GPUS} path_to_script.py --all_arguments_of_the_script

I’m wondering if it’s still exchangeable with the deepspeed launcher, of it not, what’s the nature of the facilitation.

2

It seems that the trainer uses accelerate to facilitate deepspeed. But when I look at the documentation, it seems that we still use deepspeed as the launcher, or the pytorch distribute

deepspeed --num_gpus=2 your_program.py <normal cl args> --deepspeed ds_config.json

or

python -m torch.distributed.launch --nproc_per_node=2 your_program.py <normal cl args>

You can still launch the script with the deepspeed and pytorch distribute command. However, since the trainer refacto, the trainer backend relies totally on accelerate. The accelerate launch command is there to simplify your life since you only need to remember one command + easy to set up the parameters using accelerate config.


But it didn’t mention using the accelerate launcher. I’m confused by this since trainer uses accelerate to facilitate the deepspeed integration.

Check the doc here! But I agree with you, this section is well hidden :sweat_smile:.

As for the TRL library, it seems that it’s using the accelerate library for it’s trainers as well, but for the TRL library, the official way to launch it is to use the accelerate launcher

TRL Trainer relies on transformers Trainer. This is why you can use accelerate there as well. We should probably update more our doc to emphasize on accelerate indeed.

I’m wondering if it’s still exchangeable with the deepspeed launcher, of it not, what’s the nature of the facilitation.

Under the hood, accelerate launch uses the deepspeed launcher. Same for pytorch distributed.

3

Thanks for the detailed explanation

Under the hood, accelerate launch uses the deepspeed launcher. Same for pytorch distributed.

Ah, that clears up a lot for me. For ‘Same for pytorch distributed.’, does that mean accelerate launch uses pytorch distributed? Or pytorch distributed uses the deepspeed launcher?

4

Yes, you can see it in this line here. Also see pytorch doc.

1 Like
5

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.