Ideally we shouldn’t do this, but if you can modify the last else statement from
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") to
device = torch.device("cuda:1" if torch.cuda.is_available() else "cpu")
in the _setup_devices function on training_args.py file in the transformer library.