Does the HF Trainer class support multi-node training?

Beginners
1

Does the HF Trainer class support multi-node training? Or only single-host multi-GPU training?

2

It supports both single-node and multi-node distributed training with the PyTorch launcher (torch.distributed.launch)

2 Likes
3

thanks! who is doing the cross-instance allreduce? PyTorch DDP? Or Horovod? or some custom HF allreduce? any sample :slight_smile: ?

4

It’s standard PyTorch DDP behind the scenes.

1 Like
5

@sgugger I tried the distributed training per this HF blog and it failed with multi-node training. It gave an mpirun error when i tried to use multiple instances (similar to when i tried the AWS blog). In both cases i tried with ml.p3.16xlarge and ml.p3dn.24xlarge instances.

Edit: I tried as written and with all variations of current PyTorch/HF URI containers (can’t link due to new account, limited to 2)

Any ideas on how to fix?

image
image2136Ă—538 69.3 KB