Hi! I’m trying to replicate the basic OPT examples from the documentation and I keep getting a CUDA is out of memory error. I tried using low_cpu_mem_usage=True since that has been a solution on other models, but it doesn’t make a difference. Code is basic: from transformers import AutoModelForCa…

OPT Memory problem

cog June 2, 2022, 2:51am 3

hi.

if your machine install multiple GPUs, load model with Data Parallel or Distributed Data parallel shall be help.

regards.

Topic		Replies	Views
Facebook/opt-30b model inferencing Models	3	2681	January 19, 2023
Running inference on OPT 30m on GPU Beginners	2	2271	May 18, 2022
Fine-tune OPT 13B: CUDA out of memory error (720gb vram, batch size 1, fp16)! Beginners	6	4589	July 25, 2022
Training out of memory 🤗Transformers	0	231	July 18, 2024
Cuda out of memory error Intermediate	11	42606	January 27, 2025