Hi Friends 
I’m training a model on two nodes (using Huggingface Trainer) and in the console, both nodes log the same loss. Is that expected or are both nodes seeing the same training data?
Yes, that is how DDP works 
2 Likes