RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CU

[saulml]

Hi All,

I am getting below error and tried pterry much everything. But did not work out. There are 8 clases in the dataset. I am not sure how to resolve the issue.

The code is:

Convert data to InputExample format

Convert labels to integers

for example in train_examples:
example.label = int(example.label)
#print(example.label)

Wrap the training examples in a SentencesDataset for compatibility with DataLoader

train_dataset = SentencesDataset(train_examples, model)
train_dataloader = DataLoader(train_dataset, shuffle=True, batch_size=16)

Define the loss function with SoftmaxLoss for binary classification

train_loss = losses.SoftmaxLoss(
model=model,
sentence_embedding_dimension=model.get_sentence_embedding_dimension(),
num_labels=8
)

Train the model using fit()

model.fit(
train_objectives=[(train_dataloader, train_loss)],
epochs=4,
warmup_steps=100
)

The error is:
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Thank you,

Seyhan

[John6666]

That error says CUDA, but it’s actually an error that often appears in places that have nothing to do with CUDA. Sometimes it can be fixed by tweaking the upper and lower limit settings.

CUDA error: device-side assert triggered

[saulml]

Thanks Pal!