CUDA Memory issue for model.generate() in AutoModelForCausalLM

[John6666]

I don’t multi-GPU, so I can’t help you solve the problem, but isn’t it similar to this problem…?

Issue with LoRA Adapter Loading on Multiple GPUs during Fine-Tuning with Accelerate and SFTTrainer

Hello Hugging Face Community, I’m working on fine-tuning a pre-trained LLaMA 3.1 model using LoRA adapters with the goal of performing additive tuning—continuing to fine-tune an existing LoRA adapter or adding a new one. I’m using the transformers, peft, trl, and accelerate libraries for this task. When I attempt to load a pre-trained LoRA adapter onto my base model and distribute the model across multiple GPUs using Accelerate , I observe that the GPU memory did not seem to be normal. I guess…

If several people are complaining of the same symptoms, it could be a bug in the library, but it’s not confirmed…

GitHub - huggingface/accelerate: 🚀 A simple way to launch, train, and use PyTorch...