Hi, I am trying to train a zero-shot topic classification model on XNLI/vi dataset using phobert-base-v2 on Google Colab Pro+. I got the following error whenever it reaches a certain steps with different batch_size: RuntimeError: CUDA error: device-side assert triggered Compile with TORCH_USE_CU…

CUDA error: device-side assert triggered after a certain steps

dSiddhesh December 29, 2023, 7:52am 6

I am getting the same error.
RuntimeError: CUDA error: device-side assert triggered
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

The same case, batch size 2, at step 32, Pretraining the llama model,
did anyone find any solution?

Topic		Replies	Views
RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CU 🤗Transformers	2	1158	November 1, 2024
[HELP] RuntimeError: CUDA error: device-side assert triggered Beginners	20	54475	October 23, 2024
CUDA error: device-side assert triggered 🤗Transformers	3	4276	June 4, 2021
CUDA error that only occurs on multiple gpus when doing batched training Beginners	0	769	June 25, 2024
RuntimeError: CUDA error: device-side assert triggered 4x10 🤗Transformers	0	177	April 11, 2024

CUDA error: device-side assert triggered after a certain steps

Related topics