Llama2 70b - Cuda out of memory exceptions

🤗Transformers
1

Hello,

I have 2 GPU of 24 GB RTX 4090 GPU.

I want to fine-tune the 70b model but it throws a cuda out of memory exceptions even though I have used Lora and BitsAndBytesConfig.

Let me know if I’m overlooking this or please give me suggestions.

Thanks.