I overlooked the fact that you were able to use one of the GPUs…
For normal inference, you don’t need to do much special configuration for multi-GPU (the accelerate library will handle it for you), but it seems that you need to manually configure various settings when training the model.
1 Like