Seq2SeqTrainer multiple GPUs

Does Seq2SeqTrainer automatically utilize every available GPU on the system without expressly setting any parameter? The per_device_train_batch_size seems to default to 8. Is there anything in particular I need to do to instruct the trainer to use every GPU available on the system?

1 Like

Cases where it is not used appropriately and evenly are often reported, but it seems that it will automatically use multiple GPUs in most cases.
If you want to specify in detail, check out PyTorch’s DDP.
Also, if I may say so, you should definitely install the accelerate library.

pip install -U accelerate
1 Like

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.