Which method is use HF Trainer with multiple GPU?

[Indramal]

According to the following question, the trainer will handle multiple GPU work. What is the method it uses?

DataParallel (DP) or TensorParallel (TP) or PipelineParallel (PP) or DPP, what?

Training using multiple GPUs

I would like to train some models to multiple GPUs. Let suppose that I use model from HF library, but I am using my own trainers,dataloader,collators etc. Where I should focus to implement multiple GPU training? I need to make changes only in the Trainer class? If yes, can you give me a brief description? Thank you in avance.

[AndreaSottana]

Based on this line of code it looks like it is using nn.DataParallel, however I haven’t fully looked at every line of the Trainer class so they may also be using other methods in other points.

[Indramal]

Thank you for your information. Old Trainer documents have to configure that. But new document doese not mention it. Old Doc - Trainer — transformers 4.7.0 documentation

[SUNM]

hi @AndreaSottana , sorry I am trying to fine tune got-neo because of the Cuda memory issue I need to use multiple GPU. I use the trainer in hugging face which I understand it will use multiple GPu . but my results are very strange and very different than when I use 1 GPU. Would you please help me how you use multiple GPU for fine tunning the model.?

Many thanks

[SUNM]

Hi @Indramal
sorry I am trying to fine tune got-neo because of the Cuda memory issue I need to use multiple GPU. I use the trainer in hugging face which I understand it will use multiple GPu . but my results are very strange and very different than when I use 1 GPU. Would you please help me how you use multiple GPU for fine tunning the model.?

Many thanks