Accelerate remain stuck on using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular devic

trix-mv · February 17, 2025, 3:45pm

Hello, i’m running with accelerate launch … a fine-tuning script, at the moment of creating the sfttrainer, I get the following warnings

Creating trainer
[rank9]:[W217 15:41:22.748155469 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 9]  using GPU 9 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank1]:[W217 15:41:24.786532587 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 1]  using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank0]:[W217 15:41:26.495752054 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 0]  using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank3]:[W217 15:41:28.604763182 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 3]  using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank13]:[W217 15:41:29.322907354 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 13]  using GPU 13 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank8]:[W217 15:41:29.432440926 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 8]  using GPU 8 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank15]:[W217 15:41:29.592298265 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 15]  using GPU 15 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank2]:[W217 15:41:29.675908289 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 2]  using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank14]:[W217 15:41:29.681576596 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 14]  using GPU 14 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank7]:[W217 15:41:29.849474231 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 7]  using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank5]:[W217 15:41:30.207243683 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 5]  using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank10]:[W217 15:41:30.317170644 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 10]  using GPU 10 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank4]:[W217 15:41:30.500803586 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 4]  using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank6]:[W217 15:41:30.619192821 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 6]  using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank12]:[W217 15:41:30.673832092 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 12]  using GPU 12 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank11]:[W217 15:41:30.795653955 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 11]  using GPU 11 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.

And the script remains stuck there, I’m not really sure on how to solve this, what can I do?

John6666 · February 17, 2025, 5:13pm

It’s not an easy problem to solve, but it seems that in some cases it can be avoided by downgrading PyTorch.

github.com/mosaicml/streaming

Issue: NCCL ProcessGroup Device Mapping Bug when using streaming with accelerate

opened 12:20PM - 18 Oct 24 UTC

closed 12:42PM - 25 Oct 24 UTC

wangyanhui666

bug

## Environment - OS: [Ubuntu 20.04] - Hardware (GPU, or instance type): [A…100] ## To reproduce Steps to reproduce the behavior: 1.Set up a distributed training pipeline using accelerate, to train a diffusion 2.Use the streaming dataset 3.Run the distributed training using NCCL backend on multiple GPUs. warning:[rank1]:[W1018 11:57:38.174478138 ProcessGroupNCCL.cpp:4115] [PG ID 1 PG GUID 0 Rank 1] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id. [rank2]:[W1018 11:57:38.183572992 ProcessGroupNCCL.cpp:4115] [PG ID 1 PG GUID 0 Rank 2] using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id." ## Expected behavior The devices should be correctly mapped for each rank, and the dist.barrier() call should execute without any issues. ## Additional context It seems the issue arises from the device mapping in the distributed context when using streaming with accelerate. In streaming dataset use dist.barrier(). But accelerate use accelerator.wait_for_everyone() to sync. however i add dist.init_process_group(backend='nccl') in my accelerate based code. Still have this warning. The devices are not properly identified by NCCL when calling dist.barrier(). A potential solution could be to ensure device_ids are specified correctly when invoking dist.barrier(), or init_process_group() needs to properly associate rank with the correct GPU.

Topic		Replies	Views
Multiple GPUs are being used despite `--num_processes 1` 🤗Accelerate	0	61	July 31, 2024
Accelerate.prepare hang on single machine multiple gpu 🤗Accelerate	3	1160	July 16, 2023
Trouble using Torchmetrics with Accelerate in stable diffusion finetuning script Beginners	2	498	January 18, 2024
How to use specific gpu in accelerate? 🤗Accelerate	10	6973	April 25, 2024
HF Accelerate uses multiple GPUs even when setting `num_processes` to 1 🤗Accelerate	0	50	August 2, 2024

Accelerate remain stuck on using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular devic

Related topics