Hi with my trainer I get an error and I am not able to fix it. Log as follows, just when 100% is reached:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 14.76 GiB total capacity; 11.39 GiB already allocated; 253.88 MiB free; 13.22 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Progress:| |: 0% 1/1600 [00:08<3:59:07, 8.97s/it, loss=0.686, lr=1e-6]
Traceback (most recent call last):
File “/usr/local/bin/accelerate”, line 8, in
sys.exit(main())
File “/usr/local/lib/python3.8/dist-packages/accelerate/commands/accelerate_cli.py”, line 43, in main
args.func(args)
File “/usr/local/lib/python3.8/dist-packages/accelerate/commands/launch.py”, line 837, in launch_command
simple_launcher(args)
File “/usr/local/lib/python3.8/dist-packages/accelerate/commands/launch.py”, line 354, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
The problem gets when Generating class images: 100%
How could I fix it? Thanks in advance