Hi,
I am performing multi-GPU training, and even with batch_size = 1, the GPU memory is almost full.
That’s why when it comes to the validation stage, I get CUDA OOM error.
Is there a way to set up one device for validation only? Like “cuda:7” for validation and others from 0 to 6 for training?
I have looked in the documentation and asked ChatGPT, but no luck.