'm doing some lora experiments on my 8-GPU machine, while im running other training process on 6 of 8 gpus. And I cant use CUDA_VISIBLE_DEVICES for i need to launch the trainging by using multi
File “/home/asasa/repo/kohya_ss/train_network.py”, line 222, in train
accelerator.device = torch.device(“cuda:7” if torch.cuda.is_available() else “cpu”)
File “/home/asasa/repo/kohya_ss/train_new.py”, line 302, in train_lora
trainer.train(args)
File “/home/asasa/repo/kohya_ss/train_new.py”, line 733, in total_train
train_args = train_lora(train_dataset_dir, name, lora_dir, log_dir)
AttributeError: can’t set attribute ‘device’
threads. If I set the env var, that would work like a two gpus distributed training. But i need train different exp on different gpu.
So I decide to set the device attribute straightly when creating the Acclerator object, but i failed for there is no params named device or sth. And then I tried set the attribute after created the Acclerator object, it fails too, like the error above.
My train process need one gpu for each, so i have to make them seperate. is there anyway to set the device before the train process?
I have no experience in actual use since I only have one crappy GPU, but it seems to be the usual practice to specify device_map if you want to use multiple GPUs at will. For more information, please search “device_map”.