Starcoder: CUDA out of memory

Beginners
1

I am trying to run

bigcode/starcoder

model on Amazon EC2. I have 8 tesla T4 GPUs with 16GB RAM, but somehow I still encounter the “Cuda out of memory” error.

What possible solutions can this issue have? Thank you.

Code Snapshot:

from transformers import AutoModelForCausalLM, AutoTokenizer
from accelerate import Accelerator
accelerator = Accelerator()
checkpoint = “bigcode/starcoder”
device = accelerator.device # for GPU usage or “cpu” for CPU usage
print(‘Reached device Selection.’)
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
print(‘Tokens generated.’)
model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
accelerator.prepare(model)
inputs = tokenizer.encode(“def print_hello_world():”, return_tensors=“pt”).to(device)
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))

Error:

Screenshot 2023-11-27 154125
Screenshot 2023-11-27 1541251922×603 55.7 KB

2

same issue here

3

Hi!
I searched the internet and it seems that accelerator.device always uses cuda:0, so the model is not distributed to the GPUs.

Maybe this helps you.

4

is it a limit for most of the LLM model which can’t scale into multiple GPUs?

5

AutoModelForCausalLM.from_pretrained has device_map option, set it to device_map="auto" it will split model layers into different available devices.

Correct:
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto").to(device)