How do I fix the "RuntimeError: CUDA error: CUDA driver version is insufficient for CUDA runtime version CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect." error?

Spaces
1

Whenever I try to run my stable diffusion space it returns this error “RuntimeError: CUDA error: CUDA driver version is insufficient for CUDA runtime version CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.” how do I fix it?

1 Like
2

Hello @AdamOswald1,

Could you provide a link to your Space along with the type of hardware you chose ?

In the meantime if your Space is torch based you can try adding this to your requirements:

--extra-index-url https://download.pytorch.org/whl/cu113
torch
3

CPU Basic

4

@AdamOswald1 the issue is during the inference a torch.Generator('cuda') is created while you are using a CPU hardware (so cuda is not available).

Similar to what is done L94 of the app.py you should test for cuda availability and if it is not available simply create the generator like this torch.Generator().

So as a solution you should either upgrade this space to a GPU hardware or replace L142 of app.py by:

if torch.cuda.is_available():
  generator = torch.Generator('cuda').manual_seed(seed)
else:
  generator = torch.Generator().manual_seed(seed)
5

Fixed now! Thanks!

6

Hey! I am running whisper on a space, and they require torch==1.13.1, which in turn deprecated support for 11.3 cuda drivers :frowning: so I have to run it on CPU upgrade… do you know a solution for my situation? Thanks!!!

7

hi @chr1 can you run a custom Docker Space for your project? thus you can choose your requirements accordingly

8

hi, thanks for helping! Yes, it is running in a custom docker space, but the torch version was set by whisper, “not me”.

9

great! did you solve the issue then?

1 Like
10

Nope… that torch version (the one whisper requires) needs >=11.7 driver version (im a newbie…could be wrong) and my understanding is that spaces use cuda driver version 11.3

11

could you share your Space? If you’re using a custom Dockerfile you can use any driver for example
FROM nvidia/cuda:11.7.1-cudnn8-devel-ubuntu22.04

Indeed, if you’re using our default Gradio/Streamlit SDKs the base image is FROM nvidia/cuda:11.3.1-cudnn8-devel-ubuntu18.04

1 Like
12

OH! thanks! I didn’t know I could choose. I’ll try and let you know. :pray:t3:

1 Like
13

Have the same error here, can you maybe explain to me how to fix that as if I am 5 years old…

1 Like