Dataloader_num_workers in a torch.distributed setup using HF Trainer

Beginners
1

Hi everyone - super quick question. I looked around and I couldn’t find this previously asked, but my apologies if I missed something!

Wondering if I have HF trainer set up using torch.distributed.launch on 8 gpus… if my dataloader_num_workers = 10… is that 10 total processes for dataloaders or 10*8=80 processes?

Thank you again!

2

That’s correct.

3

Sorry… which one… 10 or 80? (Is it num workers * n_gpu or num_workers?) thank you!!

4

10 * 8 = 80

5

thank you!!!