Hi All, Trying to use gemma with BitsAndBytes config for quanitzation. Then further wrapping in pytorch lighning module, and inferencing with ddp strategy continiously getting error of: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! …

BitsAndBytes With DDP

John6666 October 6, 2024, 12:41am 2

maybe.

model = gem(model = gemma, processor = processor, config = config)
model.to("cuda")

Topic		Replies	Views
Tensor not on the same device after using BitsandBytesConfig 🤗Transformers	0	106	June 27, 2024
BitsAndBytesConfig is not compitable in TPU env 🤗Transformers	2	245	July 6, 2024
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:2 and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select) DeepSpeed	5	3474	August 26, 2024
Multi-gpu inference llama-3.2 vision with QLoRA 🤗Accelerate	4	140	April 25, 2025
BitsandBytes conflict with Accelerate 🤗Accelerate	6	784	April 14, 2025