Does one need to load the model to GPU before calling train when using accelerate?

Beginners
1

Do I need to call:

        device = torch.device(f"cuda:{0}" if torch.cuda.is_available() else "cpu")
        model = model.to(device)

if I am using accelerate? e.g.,

accelerate launch --config_file hf_training.py

I was worried about the accelerator.prepare(...) not working as expected.

related question: huggingface transformers - How to adapt LLaMA v2 model to less than 7B parameters? - Stack Overflow

cross: huggingface transformers - Does one need to load the model to GPU before calling train when using accelerate? - Stack Overflow

2

HF always loads your model to GPU if it’s available. With accelerate it does it too.

The Trainer checks torch.cuda.is_available() to detect if GPUs are available.

ref https://github.com/huggingface/transformers/blob/bffac926ca6bc6c965a92bfbfd00c567a2c0fb90/src/transformers/trainer.py#L2298

so

       # from accelerate import Accelerator
        # accelerator = Accelerator()
        # # self.is_deepspeed_enabled = getattr(accelerator.state, "deepspeed_plugin", None) is not None
        # is_fsdp_enabled = getattr(accelerator.state, "fsdp_plugin", None) is not None
        # if not is_fsdp_enabled: # not sure if this is needed but its for sure safer
        #     # maybe figuring out how to run everything with accelerate would fix things...
        #     # ref: https://stackoverflow.com/questions/77204403/does-one-need-to-load-the-model-to-gpu-before-calling-train-when-using-accelerat
        #     device = torch.device(f"cuda:{0}" if torch.cuda.is_available() else "cpu")
        #     model = model.to(device)

is not needed. Just load the model and trainer will take care of it.

3

related HF discord: Discord