Hi everyone, Thanks in advance for helping! First I just have to say, as my first post here, Huggingface is awesome. We have been using the tools/libraries for a while for NLP work and it is just a pleasure to use and so applicable to real-world problems!!! We are “graduating” if you will from sin…

The Trainer code will run on distributed or one GPU without any change. Regarding your other questions: you need to define your model in all processes, they will see different part of the data each and all copies will be kept the same. the tokenizer and dataset preprocessing can either be done on …

Boilerplate for Trainer using torch.distributed

Beginners

mohotmoz January 11, 2022, 3:37pm 5

that is exactly what i needed to hear - thank you so much for your reply!!!

Topic		Replies	Views
How to run an end to end example of distributed data parallel with hugging face's trainer api (ideally on a single node multiple gpus)? Intermediate	17	18039	September 6, 2023
How to run single-node, multi-GPU training with HF Trainer? 🤗Transformers	5	15302	October 16, 2024
Multi gpu training 🤗Transformers	3	6029	April 24, 2022
Training using multiple GPUs Beginners	20	20149	February 25, 2024
Single Node Multi GPU FlanT5 fine-tuning using HF Dataset and HF Trainer 🤗Transformers	4	2066	July 5, 2023

Boilerplate for Trainer using torch.distributed

Related topics