Hey everybody,
for my masters thesis I’m currently trying to run class conditional diffusion on microscopy images.
For this I need images with a resolution of 512x512, so I’m relying on a compute cluster provided by my university. Training on 1 GPU results in an epoch time of 32-45min, which is not at all doable for me. But I can’t seem to get Multi-GPU working correctly. Following are my specs:
- `Accelerate` version: 0.21.0.dev0
- Platform: Linux-3.10.0-1160.83.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.16
- Numpy version: 1.24.3
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- PyTorch XPU available: False
- System RAM: 355.40 GB
- GPU type: Quadro P6000
- `Accelerate` default config:
- compute_environment: LOCAL_MACHINE
- distributed_type: MULTI_GPU
- mixed_precision: fp16
- use_cpu: False
- num_processes: 4
- machine_rank: 0
- num_machines: 1
- gpu_ids: 0,1,2,3
- rdzv_backend: static
- same_network: True
- main_training_function: main
- downcast_bf16: no
- tpu_use_cluster: False
- tpu_use_sudo: False
- tpu_env: []
Docker version 23.0.1, build a5ee5b1
GPUS: 10x Quadro P6000 24GB`
And this is my error message, when I try to run `accelerate test`:
`
Running: accelerate-launch /opt/conda/envs/accelerate/lib/python3.8/site-packages/accelerate/test_utils/scripts/test_script.py
stdout: [2023-07-13 20:19:50,932] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
stdout: [2023-07-13 20:19:55,740] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
stdout: [2023-07-13 20:19:55,745] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
stdout: [2023-07-13 20:19:55,758] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
stdout: [2023-07-13 20:19:55,763] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
stdout: Distributed environment: MULTI_GPU Backend: nccl
stdout: Num processes: 4
stdout: Process index: 3
stdout: Local process index: 3
stdout: Device: cuda:3
stdout:
stdout: Mixed precision type: fp16
stdout:
stdout: **Initialization**
stdout: Testing, testing. 1, 2, 3.
stdout: Distributed environment: MULTI_GPU Backend: nccl
stdout: Num processes: 4
stdout: Process index: 0
stdout: Local process index: 0
stdout: Device: cuda:0
stdout:
stdout: Mixed precision type: fp16
stdout:
stdout: Distributed environment: MULTI_GPU Backend: nccl
stdout: Num processes: 4
stdout: Process index: 1
stdout: Local process index: 1
stdout: Device: cuda:1
stdout:
stdout: Mixed precision type: fp16
stdout:
stdout: Distributed environment: MULTI_GPU Backend: nccl
stdout: Num processes: 4
stdout: Process index: 2
stdout: Local process index: 2
stdout: Device: cuda:2
stdout:
stdout: Mixed precision type: fp16
stdout:
stdout: [20:20:03] ERROR failed (exitcode: -7) local_rank: 0 (pid: 927) of binary: /opt/conda/envs/accelerate/bin/python3 api.py:672
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /opt/conda/envs/accelerate/bin/accelerate:8 in <module> │
│ │
│ 5 from accelerate.commands.accelerate_cli import main │
│ 6 if __name__ == '__main__': │
│ 7 │ sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?
```, '', sys.argv[0]) │
│ ❱ 8 │ sys.exit(main()) │
│ 9 │
│ │
│ /opt/conda/envs/accelerate/lib/python3.8/site-packages/accelerate/commands/accelerate_cli.py:45 │
│ in main │
│ │
│ 42 │ │ exit(1) │
│ 43 │ │
│ 44 │ # Run │
│ ❱ 45 │ args.func(args) │
│ 46 │
│ 47 │
│ 48 if __name__ == "__main__": │
│ │
│ /opt/conda/envs/accelerate/lib/python3.8/site-packages/accelerate/commands/test.py:54 in │
│ test_command │
│ │
│ 51 │ │ test_args = f"--config_file={args.config_file} {script_name}" │
│ 52 │ │
│ 53 │ cmd = ["accelerate-launch"] + test_args.split() │
│ ❱ 54 │ result = execute_subprocess_async(cmd, env=os.environ.copy()) │
│ 55 │ if result.returncode == 0: │
│ 56 │ │ print("Test is a success! You are ready for your distributed training!") │
│ 57 │
│ │
│ /opt/conda/envs/accelerate/lib/python3.8/site-packages/accelerate/test_utils/testing.py:383 in │
│ execute_subprocess_async │
│ │
│ 380 │ cmd_str = " ".join(cmd) │
│ 381 │ if result.returncode > 0: │
│ 382 │ │ stderr = "\n".join(result.stderr) │
│ ❱ 383 │ │ raise RuntimeError( │
│ 384 │ │ │ f"'{cmd_str}' failed with returncode {result.returncode}\n\n" │
│ 385 │ │ │ f"The combined stderr from workers follows:\n{stderr}" │
│ 386 │ │ ) │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: 'accelerate-launch /opt/conda/envs/accelerate/lib/python3.8/site-packages/accelerate/test_utils/scripts/test_script.py' failed with returncode 1
The combined stderr from workers follows:
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /opt/conda/envs/accelerate/bin/accelerate-launch:8 in <module> │
│ │
│ 5 from accelerate.commands.launch import main │
│ 6 if __name__ == '__main__': │
│ 7 │ sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?
```, '', sys.argv[0]) │
│ ❱ 8 │ sys.exit(main()) │
│ 9 │
│ │
│ /opt/conda/envs/accelerate/lib/python3.8/site-packages/accelerate/commands/launch.py:975 in main │
│ │
│ 972 def main(): │
│ 973 │ parser = launch_command_parser() │
│ 974 │ args = parser.parse_args() │
│ ❱ 975 │ launch_command(args) │
│ 976 │
│ 977 │
│ 978 if __name__ == "__main__": │
│ │
│ /opt/conda/envs/accelerate/lib/python3.8/site-packages/accelerate/commands/launch.py:960 in │
│ launch_command │
│ │
│ 957 │ elif args.use_megatron_lm and not args.cpu: │
│ 958 │ │ multi_gpu_launcher(args) │
│ 959 │ elif args.multi_gpu and not args.cpu: │
│ ❱ 960 │ │ multi_gpu_launcher(args) │
│ 961 │ elif args.tpu and not args.cpu: │
│ 962 │ │ if args.tpu_use_cluster: │
│ 963 │ │ │ tpu_pod_launcher(args) │
│ │
│ /opt/conda/envs/accelerate/lib/python3.8/site-packages/accelerate/commands/launch.py:649 in │
│ multi_gpu_launcher │
│ │
│ 646 │ ) │
│ 647 │ with patch_environment(**current_env): │
│ 648 │ │ try: │
│ ❱ 649 │ │ │ distrib_run.run(args) │
│ 650 │ │ except Exception: │
│ 651 │ │ │ if is_rich_available() and debug: │
│ 652 │ │ │ │ console = get_console() │
│ │
│ /opt/conda/envs/accelerate/lib/python3.8/site-packages/torch/distributed/run.py:785 in run │
│ │
│ 782 │ │ ) │
│ 783 │ │
│ 784 │ config, cmd, cmd_args = config_from_args(args) │
│ ❱ 785 │ elastic_launch( │
│ 786 │ │ config=config, │
│ 787 │ │ entrypoint=cmd, │
│ 788 │ )(*cmd_args) │
│ │
│ /opt/conda/envs/accelerate/lib/python3.8/site-packages/torch/distributed/launcher/api.py:134 in │
│ __call__ │
│ │
│ 131 │ │ self._entrypoint = entrypoint │
│ 132 │ │
│ 133 │ def __call__(self, *args): │
│ ❱ 134 │ │ return launch_agent(self._config, self._entrypoint, list(args)) │
│ 135 │
│ 136 │
│ 137 def _get_entrypoint_name( │
│ │
│ /opt/conda/envs/accelerate/lib/python3.8/site-packages/torch/distributed/launcher/api.py:250 in │
│ launch_agent │
│ │
│ 247 │ │ │ # if the error files for the failed children exist │
│ 248 │ │ │ # @record will copy the first error (root cause) │
│ 249 │ │ │ # to the error file of the launcher process. │
│ ❱ 250 │ │ │ raise ChildFailedError( │
│ 251 │ │ │ │ name=entrypoint_name, │
│ 252 │ │ │ │ failures=result.failures, │
│ 253 │ │ │ ) │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ChildFailedError:
============================================================
/opt/conda/envs/accelerate/lib/python3.8/site-packages/accelerate/test_utils/scripts/test_script.py FAILED
------------------------------------------------------------
Failures:
[1]:
time : 2023-07-13_20:20:03
host : d412dfc663fe
rank : 1 (local_rank: 1)
exitcode : -7 (pid: 928)
error_file: <N/A>
traceback : Signal 7 (SIGBUS) received by PID 928
[2]:
time : 2023-07-13_20:20:03
host : d412dfc663fe
rank : 2 (local_rank: 2)
exitcode : -7 (pid: 929)
error_file: <N/A>
traceback : Signal 7 (SIGBUS) received by PID 929
[3]:
time : 2023-07-13_20:20:03
host : d412dfc663fe
rank : 3 (local_rank: 3)
exitcode : -7 (pid: 930)
error_file: <N/A>
traceback : Signal 7 (SIGBUS) received by PID 930
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2023-07-13_20:20:03
host : d412dfc663fe
rank : 0 (local_rank: 0)
exitcode : -7 (pid: 927)
error_file: <N/A>
traceback : Signal 7 (SIGBUS) received by PID 927
============================================================
ERROR conda.cli.main_run:execute(47): `conda run accelerate test` failed. (See above for error)
Any help would be greatly appreciated!