I have a 3 GPU config 1x4080 and 2x3090. The 3090s are connected via NVLink, tested it with NCCL test all works fine. I have an Accelerate config where I set to use only the 2 3090s. When I run accelerate test :
- Output “Using RTX 4000 series which doesn’t support faster communication speedups. Ensuring P2P and IB communications are disabled.” even is I’ve specified to use only the 2 3090s. Does it always “looks” at all the GPUs not only the ones set to use?
- I have NCCL_DEBUG set to INFO and see “NCCL_P2P_LEVEL set by environment to LOC” though I’d like to use ‘NVL’. It’s not clear for me how can I configure this with Accelerate. I have export NCCL_P2P_LEVEL=NVL but seemingly has no effect.
How can I configure Accelerate to use NVL?
stdout: debian:34193:34210 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC
stdout: debian:34193:34210 [1] NCCL INFO === System : maxBw 6.0 totalBw 48.0 ===
stdout: debian:34193:34210 [1] NCCL INFO CPU/FFFFFFFFFFFFFFFF (1/2/-1)
stdout: debian:34193:34210 [1] NCCL INFO + PCI[12.0] - GPU/B000 (1)
stdout: debian:34193:34210 [1] NCCL INFO + NVL[48.0] - GPU/A000
stdout: debian:34193:34210 [1] NCCL INFO + PCI[6.0] - GPU/A000 (0)
stdout: debian:34193:34210 [1] NCCL INFO + NVL[48.0] - GPU/B000