Hi! I’m trying to finetune a huggingface model using the Lightning trainer and Deepspeed. However, when I use multiple gpus I get the following error: RuntimeError: a leaf Variable that requires grad is being used in an in-place operation. Can someone help me solving this issue? The code I am us…

DeepSpeed error: a leaf Variable that requires grad is being used in an in-place operation

SSamDav July 26, 2024, 3:27pm 2

I only get this error when I use

 process_group_backend="gloo",

Topic		Replies	Views
Gpt-neo inference with Deepspeed: IndexError: Dimension out of range Beginners	0	483	August 10, 2021
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:2 and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select) DeepSpeed	5	3473	August 26, 2024
Question about using trainer with DeepSpeed 🤗Transformers	0	462	April 25, 2023
How to use trainer with deepspeed Beginners	0	342	January 12, 2024
[Deepspeed] ZeRO-Infinity integration released and config changes DeepSpeed	2	2302	April 28, 2021