@SUNM It is my understanding that if you train a model with the huggingface Trainer class (as you’ve shown in the post Finetuning GPT2 using Multiple GPU and Trainer) and you have multiple GPUs available on your system, then Trainer will use all of the available GPUs unless explicitly told not to do so.
So, if you have 4 GPUs available but you set the environment variable CUDA_VISIBLE_DEVICES=“1,2,3”, then Trainer will only utilize those 3 GPU instead of all 4 of them.