Hi, I am trying to train a zero-shot topic classification model on XNLI/vi dataset using phobert-base-v2 on Google Colab Pro+. I got the following error whenever it reaches a certain steps with different batch_size: RuntimeError: CUDA error: device-side assert triggered Compile with TORCH_USE_CU…

CUDA error: device-side assert triggered after a certain steps

ankitb131313 January 31, 2024, 3:04am 7

I was able to get past this error by setting PYTORCH_USE_CUDA_DSA as “1”

os.environ["PYTORCH_USE_CUDA_DSA"] = "1"

torch version is 2.2.0

Topic		Replies	Views
RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CU 🤗Transformers	2	1158	November 1, 2024
[HELP] RuntimeError: CUDA error: device-side assert triggered Beginners	20	54475	October 23, 2024
CUDA error: device-side assert triggered 🤗Transformers	3	4276	June 4, 2021
CUDA error that only occurs on multiple gpus when doing batched training Beginners	0	769	June 25, 2024
RuntimeError: CUDA error: device-side assert triggered 4x10 🤗Transformers	0	177	April 11, 2024